Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: emit files on demand and fix racy emit #15220

Merged
merged 32 commits into from
Jul 19, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
f4ebdfb
Start of work.
dsherret Jul 13, 2022
1ac4ca0
Merge branch 'main' into lazy_emit
dsherret Jul 13, 2022
6e9707b
More changes.
dsherret Jul 13, 2022
6a0c191
Fixes.
dsherret Jul 13, 2022
250722f
Clippy.
dsherret Jul 13, 2022
85fc0be
Verify the source hash in the cache in order to encourage consumers t…
dsherret Jul 13, 2022
76adfda
Merge branch 'main' into lazy_emit
dsherret Jul 13, 2022
a4e0615
Fix coverage code.
dsherret Jul 14, 2022
e83113b
Emit cache unit tests
dsherret Jul 14, 2022
3ae3431
Merge branch 'main' into lazy_emit
dsherret Jul 14, 2022
ceede09
Reduce memory usage by storing source map separately.
dsherret Jul 14, 2022
adecc49
Lint.
dsherret Jul 14, 2022
87fb86e
Updates.
dsherret Jul 14, 2022
098f5b8
Merge branch 'main' into lazy_emit
dsherret Jul 14, 2022
43c090b
Remove logic for optional source hash as it's now unnecessary.
dsherret Jul 14, 2022
a137bea
Merge branch 'main' into lazy_emit
dsherret Jul 14, 2022
2a131c7
Merge branch 'main' into lazy_emit
dsherret Jul 14, 2022
b01b98f
Merge branch 'main' into lazy_emit
dsherret Jul 16, 2022
0d53b1d
This should actually be the opposite way
dsherret Jul 16, 2022
4fb035e
Hash -> Hasher
dsherret Jul 16, 2022
f57d544
Alternative way of doing this.
dsherret Jul 17, 2022
a6acd76
This doesn't need to be public.
dsherret Jul 17, 2022
5f48205
Merge branch 'main' into alternative_lazy_emit
dsherret Jul 18, 2022
6f1d7bd
Updates.
dsherret Jul 18, 2022
a8d5179
Merge branch 'main' into alternative_lazy_emit
dsherret Jul 18, 2022
b46117f
Reduce emits by using the found url.
dsherret Jul 18, 2022
69a4637
Include specifier in message.
dsherret Jul 18, 2022
39bba4a
Add doc string.
dsherret Jul 18, 2022
12c2e57
Remove logging.
dsherret Jul 18, 2022
dc78d55
Clipiteedee
dsherret Jul 18, 2022
4e90d7c
Ensure `deno cache` emits
dsherret Jul 18, 2022
8b32ab3
Clippy, as per usual
dsherret Jul 18, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions cli/args/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,11 @@ impl CliOptions {
self.flags.enable_testing_features
}

/// If the --inspect or --inspect-brk flags are used.
pub fn is_inspecting(&self) -> bool {
self.flags.inspect.is_some() || self.flags.inspect_brk.is_some()
}

pub fn inspect_brk(&self) -> Option<SocketAddr> {
self.flags.inspect_brk
}
Expand Down
6 changes: 3 additions & 3 deletions cli/cache/check.rs
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ impl TypeCheckCache {
Err(err) => {
log::debug!(
concat!(
"Failed creating internal type checking cache. ",
"Failed loading internal type checking cache. ",
"Recreating...\n\nError details:\n{:#}",
),
err
Expand All @@ -35,7 +35,7 @@ impl TypeCheckCache {
Err(err) => {
log::debug!(
concat!(
"Unable to create internal cache for type checking. ",
"Unable to load internal cache for type checking. ",
"This will reduce the performance of type checking.\n\n",
"Error details:\n{:#}",
),
Expand Down Expand Up @@ -233,7 +233,7 @@ mod test {
cache.set_tsbuildinfo(&specifier1, "test");
assert_eq!(cache.get_tsbuildinfo(&specifier1), Some("test".to_string()));

// recreating the cache should not remove the data because the CLI version and state hash is the same
// recreating the cache should not remove the data because the CLI version is the same
let conn = cache.0.unwrap();
let cache =
TypeCheckCache::from_connection(conn, "2.0.0".to_string()).unwrap();
Expand Down
35 changes: 28 additions & 7 deletions cli/cache/common.rs
Original file line number Diff line number Diff line change
@@ -1,16 +1,37 @@
// Copyright 2018-2022 the Deno authors. All rights reserved. MIT license.

use std::hash::Hasher;

use deno_core::error::AnyError;
use deno_runtime::deno_webstorage::rusqlite::Connection;

/// Very fast non-cryptographically secure hash.
pub fn fast_insecure_hash(bytes: &[u8]) -> u64 {
use std::hash::Hasher;
use twox_hash::XxHash64;
/// A very fast insecure hasher that uses the xxHash algorithm.
#[derive(Default)]
pub struct FastInsecureHasher(twox_hash::XxHash64);

impl FastInsecureHasher {
pub fn new() -> Self {
Self::default()
}

pub fn write_str(&mut self, text: &str) -> &mut Self {
self.write(text.as_bytes());
self
}

pub fn write(&mut self, bytes: &[u8]) -> &mut Self {
self.0.write(bytes);
self
}

pub fn write_u64(&mut self, value: u64) -> &mut Self {
self.0.write_u64(value);
self
}

let mut hasher = XxHash64::default();
hasher.write(bytes);
hasher.finish()
pub fn finish(&self) -> u64 {
self.0.finish()
}
}

/// Runs the common sqlite pragma.
Expand Down
78 changes: 0 additions & 78 deletions cli/cache/disk_cache.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,6 @@
use crate::fs_util;
use crate::http_cache::url_to_filename;

use super::CacheType;
use super::Cacher;
use super::EmitMetadata;

use deno_ast::ModuleSpecifier;
use deno_core::error::AnyError;
use deno_core::serde_json;
use deno_core::url::Host;
use deno_core::url::Url;
use std::ffi::OsStr;
Expand Down Expand Up @@ -154,77 +147,6 @@ impl DiskCache {
fs_util::atomic_write_file(&path, data, crate::http_cache::CACHE_PERM)
.map_err(|e| with_io_context(&e, format!("{:#?}", &path)))
}

fn get_emit_metadata(
&self,
specifier: &ModuleSpecifier,
) -> Option<EmitMetadata> {
let filename = self.get_cache_filename_with_extension(specifier, "meta")?;
let bytes = self.get(&filename).ok()?;
serde_json::from_slice(&bytes).ok()
}

fn set_emit_metadata(
&self,
specifier: &ModuleSpecifier,
data: EmitMetadata,
) -> Result<(), AnyError> {
let filename = self
.get_cache_filename_with_extension(specifier, "meta")
.unwrap();
let bytes = serde_json::to_vec(&data)?;
self.set(&filename, &bytes).map_err(|e| e.into())
}
}

// todo(13302): remove and replace with sqlite database
impl Cacher for DiskCache {
fn get(
&self,
cache_type: CacheType,
specifier: &ModuleSpecifier,
) -> Option<String> {
let extension = match cache_type {
CacheType::Emit => "js",
CacheType::SourceMap => "js.map",
CacheType::Version => {
return self.get_emit_metadata(specifier).map(|d| d.version_hash)
}
};
let filename =
self.get_cache_filename_with_extension(specifier, extension)?;
self
.get(&filename)
.ok()
.and_then(|b| String::from_utf8(b).ok())
}

fn set(
&self,
cache_type: CacheType,
specifier: &ModuleSpecifier,
value: String,
) -> Result<(), AnyError> {
let extension = match cache_type {
CacheType::Emit => "js",
CacheType::SourceMap => "js.map",
CacheType::Version => {
let data = if let Some(mut data) = self.get_emit_metadata(specifier) {
data.version_hash = value;
data
} else {
EmitMetadata {
version_hash: value,
}
};
return self.set_emit_metadata(specifier, data);
}
};
let filename = self
.get_cache_filename_with_extension(specifier, extension)
.unwrap();
self.set(&filename, value.as_bytes()).map_err(|e| e.into())
}
}

#[cfg(test)]
Expand Down
Loading