Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't create page bundle as a result of a dangling resource #10386

Closed
bep opened this issue Oct 22, 2022 · 2 comments
Closed

Don't create page bundle as a result of a dangling resource #10386

bep opened this issue Oct 22, 2022 · 2 comments
Assignees
Milestone

Comments

@bep
Copy link
Member

bep commented Oct 22, 2022

Given a fictional site with languages en, jp and sv there are currently 2 conventions re. bundles and their resources vs languages.

Given this example:

content
└── mybundle
    ├── data.en.json
    ├── data.jp.json
    ├── data.sv.json
    ├── index.en.md
    ├── index.jp.md
    └── thumb.jpg

Hugo will:

  1. Return the same logical set of resources (thumb.jpg and data.json) when calling $mybundle.Resources regardless of language.
  2. Will create the missing sv version of mybundle to make data.sv.json navigable.

I'm in the process of simplifying/improving the above to enable some cool stuff, and with that I had a test failing for the 2) case above which I, even after thinking about it for a day, find surprising. I do agree that we need some better setup for filling in missing translations (e.g. lang={en,sv,jp}), but the behaviour in 2) isn't a great way to do this. Also, and this is the most important to me at the moment, it makes what I'm about to do really clumsy and hard to implement.

Note that this is only about the 2) bullet above.

@bep bep added the Proposal label Oct 22, 2022
@bep bep self-assigned this Oct 22, 2022
@bep bep added this to the v0.105.0 milestone Oct 22, 2022
bep added a commit to bep/hugo that referenced this issue Oct 24, 2022
TODO(bep) improve commit message.

Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage.

One failing example would be loading lots of big JSON data files and unmarshal them via `transform.Unmarshal`.

This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory.

Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the `HUGO_MEMORYLIMIT` environment variable (a float value representing Gigabytes).

A natural next step after this would be to use this cache for `.Content`.

Fixes gohugoio#10386
Fixes gohugoio#8307
Fixes gohugoio#8498
Fixes gohugoio#8927
Fixes gohugoio#9192
Fixes gohugoio#9189
Fixes gohugoio#7425
Fixes gohugoio#7437
Fixes gohugoio#7436
Fixes gohugoio#7882
Updates gohugoio#7544
Fixes gohugoio#9224
Fixes gohugoio#9324
Fixes gohugoio#9352
Fixes gohugoio#9343
Fixes gohugoio#9171
Fixes gohugoio#10104
Fixes gohugoio#10380
@bep bep modified the milestones: v0.105.0, v0.106.0 Oct 26, 2022
bep added a commit to bep/hugo that referenced this issue Nov 4, 2022
TODO(bep) improve commit message.

Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage.

One failing example would be loading lots of big JSON data files and unmarshal them via `transform.Unmarshal`.

This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory.

Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the `HUGO_MEMORYLIMIT` environment variable (a float value representing Gigabytes).

A natural next step after this would be to use this cache for `.Content`.

Fixes gohugoio#10386
Fixes gohugoio#8307
Fixes gohugoio#8498
Fixes gohugoio#8927
Fixes gohugoio#9192
Fixes gohugoio#9189
Fixes gohugoio#7425
Fixes gohugoio#7437
Fixes gohugoio#7436
Fixes gohugoio#7882
Updates gohugoio#7544
Fixes gohugoio#9224
Fixes gohugoio#9324
Fixes gohugoio#9352
Fixes gohugoio#9343
Fixes gohugoio#9171
Fixes gohugoio#10104
Fixes gohugoio#10380
bep added a commit to bep/hugo that referenced this issue Nov 6, 2022
TODO(bep) improve commit message.

Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage.

One failing example would be loading lots of big JSON data files and unmarshal them via `transform.Unmarshal`.

This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory.

Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the `HUGO_MEMORYLIMIT` environment variable (a float value representing Gigabytes).

A natural next step after this would be to use this cache for `.Content`.

Fixes gohugoio#10386
Fixes gohugoio#8307
Fixes gohugoio#8498
Fixes gohugoio#8927
Fixes gohugoio#9192
Fixes gohugoio#9189
Fixes gohugoio#7425
Fixes gohugoio#7437
Fixes gohugoio#7436
Fixes gohugoio#7882
Updates gohugoio#7544
Fixes gohugoio#9224
Fixes gohugoio#9324
Fixes gohugoio#9352
Fixes gohugoio#9343
Fixes gohugoio#9171
Fixes gohugoio#10104
Fixes gohugoio#10380
@bep bep modified the milestones: v0.106.0, v0.107.0 Nov 18, 2022
bep added a commit to bep/hugo that referenced this issue Nov 19, 2022
TODO(bep) improve commit message.

Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage.

One failing example would be loading lots of big JSON data files and unmarshal them via `transform.Unmarshal`.

This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory.

Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the `HUGO_MEMORYLIMIT` environment variable (a float value representing Gigabytes).

A natural next step after this would be to use this cache for `.Content`.

Fixes gohugoio#10386
Fixes gohugoio#8307
Fixes gohugoio#8498
Fixes gohugoio#8927
Fixes gohugoio#9192
Fixes gohugoio#9189
Fixes gohugoio#7425
Fixes gohugoio#7437
Fixes gohugoio#7436
Fixes gohugoio#7882
Updates gohugoio#7544
Fixes gohugoio#9224
Fixes gohugoio#9324
Fixes gohugoio#9352
Fixes gohugoio#9343
Fixes gohugoio#9171
Fixes gohugoio#10104
Fixes gohugoio#10380
bep added a commit to bep/hugo that referenced this issue Nov 19, 2022
TODO(bep) improve commit message.

Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage.

One failing example would be loading lots of big JSON data files and unmarshal them via `transform.Unmarshal`.

This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory.

Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the `HUGO_MEMORYLIMIT` environment variable (a float value representing Gigabytes).

A natural next step after this would be to use this cache for `.Content`.

Fixes gohugoio#10386
Fixes gohugoio#8307
Fixes gohugoio#8498
Fixes gohugoio#8927
Fixes gohugoio#9192
Fixes gohugoio#9189
Fixes gohugoio#7425
Fixes gohugoio#7437
Fixes gohugoio#7436
Fixes gohugoio#7882
Updates gohugoio#7544
Fixes gohugoio#9224
Fixes gohugoio#9324
Fixes gohugoio#9352
Fixes gohugoio#9343
Fixes gohugoio#9171
Fixes gohugoio#10104
Fixes gohugoio#10380
bep added a commit to bep/hugo that referenced this issue Nov 21, 2022
TODO(bep) improve commit message.

Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage.

One failing example would be loading lots of big JSON data files and unmarshal them via `transform.Unmarshal`.

This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory.

Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the `HUGO_MEMORYLIMIT` environment variable (a float value representing Gigabytes).

A natural next step after this would be to use this cache for `.Content`.

Fixes gohugoio#10386
Fixes gohugoio#8307
Fixes gohugoio#8498
Fixes gohugoio#8927
Fixes gohugoio#9192
Fixes gohugoio#9189
Fixes gohugoio#7425
Fixes gohugoio#7437
Fixes gohugoio#7436
Fixes gohugoio#7882
Updates gohugoio#7544
Fixes gohugoio#9224
Fixes gohugoio#9324
Fixes gohugoio#9352
Fixes gohugoio#9343
Fixes gohugoio#9171
Fixes gohugoio#10104
Fixes gohugoio#10380
bep added a commit to bep/hugo that referenced this issue Nov 25, 2022
TODO(bep) improve commit message.

Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage.

One failing example would be loading lots of big JSON data files and unmarshal them via `transform.Unmarshal`.

This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory.

Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the `HUGO_MEMORYLIMIT` environment variable (a float value representing Gigabytes).

A natural next step after this would be to use this cache for `.Content`.

Fixes gohugoio#10386
Fixes gohugoio#8307
Fixes gohugoio#8498
Fixes gohugoio#8927
Fixes gohugoio#9192
Fixes gohugoio#9189
Fixes gohugoio#7425
Fixes gohugoio#7437
Fixes gohugoio#7436
Fixes gohugoio#7882
Updates gohugoio#7544
Fixes gohugoio#9224
Fixes gohugoio#9324
Fixes gohugoio#9352
Fixes gohugoio#9343
Fixes gohugoio#9171
Fixes gohugoio#10104
Fixes gohugoio#10380
@bep bep modified the milestones: v0.107.0, v0.108.0 Dec 3, 2022
@bep bep modified the milestones: v0.108.0, v0.109.0 Dec 14, 2022
@bep bep modified the milestones: v0.109.0, v0.111.0, v0.110.0 Jan 26, 2023
@bep bep modified the milestones: v0.111.0, v0.112.0 Feb 15, 2023
@bep bep modified the milestones: v0.112.0, v0.113.0 Apr 15, 2023
bep added a commit to bep/hugo that referenced this issue Jun 1, 2023
TODO(bep) improve commit message.

Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage.

One failing example would be loading lots of big JSON data files and unmarshal them via `transform.Unmarshal`.

This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory.

Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the `HUGO_MEMORYLIMIT` environment variable (a float value representing Gigabytes).

A natural next step after this would be to use this cache for `.Content`.

Fixes gohugoio#10386
Fixes gohugoio#8307
Fixes gohugoio#8498
Fixes gohugoio#8927
Fixes gohugoio#9192
Fixes gohugoio#9189
Fixes gohugoio#7425
Fixes gohugoio#7437
Fixes gohugoio#7436
Fixes gohugoio#7882
Updates gohugoio#7544
Fixes gohugoio#9224
Fixes gohugoio#9324
Fixes gohugoio#9352
Fixes gohugoio#9343
Fixes gohugoio#9171
Fixes gohugoio#10104
Fixes gohugoio#10380
@bep bep removed this from the v0.113.0 milestone Jun 13, 2023
@bep bep added this to the v0.115.0 milestone Jun 13, 2023
@bep bep modified the milestones: v0.115.0, v0.116.0 Jun 30, 2023
bep added a commit to bep/hugo that referenced this issue Jul 18, 2023
TODO(bep) improve commit message.

Hugo has always been a active user of in-memory caches, but before this commit we did nothing to control the memory usage.

One failing example would be loading lots of big JSON data files and unmarshal them via `transform.Unmarshal`.

This commit consolidates all these caches into one single LRU cache with an eviction strategy that also considers used vs. available memory.

Hugo will try to limit its memory usage to 1/4 or total system memory, but this can be controlled with the `HUGO_MEMORYLIMIT` environment variable (a float value representing Gigabytes).

A natural next step after this would be to use this cache for `.Content`.

Fixes gohugoio#10386
Fixes gohugoio#8307
Fixes gohugoio#8498
Fixes gohugoio#8927
Fixes gohugoio#9192
Fixes gohugoio#9189
Fixes gohugoio#7425
Fixes gohugoio#7437
Fixes gohugoio#7436
Fixes gohugoio#7882
Updates gohugoio#7544
Fixes gohugoio#9224
Fixes gohugoio#9324
Fixes gohugoio#9352
Fixes gohugoio#9343
Fixes gohugoio#9171
Fixes gohugoio#10104
Fixes gohugoio#10380
@bep bep modified the milestones: v0.116.0, v0.117.0 Aug 1, 2023
@bep bep modified the milestones: v0.117.0, v0.118.0 Aug 30, 2023
@bep bep modified the milestones: v0.118.0, v0.119.0 Sep 15, 2023
@bep bep modified the milestones: v0.119.0, v0.120.0 Oct 5, 2023
@bep bep modified the milestones: v0.120.0, v0.121.0 Oct 31, 2023
@bep bep modified the milestones: v0.121.0, v0.122.0 Dec 6, 2023
@bep bep modified the milestones: v0.122.0, v0.123.0, v0.124.0 Jan 27, 2024
@bep
Copy link
Member Author

bep commented Jan 31, 2024

This if fixed.

@bep bep closed this as completed Jan 31, 2024
@bep bep modified the milestones: v0.124.0, v0.123.0 Jan 31, 2024
Copy link

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 22, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant