Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add in-memory queue option #77

Closed
ninjabear opened this issue May 5, 2017 · 1 comment
Closed

Add in-memory queue option #77

ninjabear opened this issue May 5, 2017 · 1 comment
Milestone

Comments

@ninjabear
Copy link
Contributor

As part of a tuned-for-throughput version of the tracker, avoiding disk accesses unlocks some extra potential.

We can achieve this by implementing the IStorage interface (this can be done dynamically). We should provide an option for this out of the box.

It'd look like this:

using Snowplow.Tracker.Storage;
using System;
using System.Collections.Generic;
using System.Linq;

...

public class MemoryStorage : IStorage
    {
        private List<string> _items = new List<string>(1000);

        public int TotalItems => _items.Count;

        public bool Delete(List<string> idList)
        {
            var ids = (from i in idList select int.Parse(i)).OrderByDescending(x=>x).ToList();
            ids.ForEach(id => _items.RemoveAt(id));
            return true;
        }

        public void Put(string item)
        {
            _items.Insert(0, item);
        }

        public List<StorageRecord> TakeLast(int n)
        {
            int totalBack = Math.Min(_items.Count(), n);
            int startIndex = _items.Count() - totalBack;

            var last = new List<StorageRecord>(totalBack);

            for (int idx=startIndex; idx<_items.Count(); idx++)
            {
                var item = new StorageRecord { Id = idx.ToString(), Item = _items[idx] };
                last.Add(item);
            }

            return last;
        }
    }
@ninjabear ninjabear added this to the Version 1.1.0 milestone May 5, 2017
@alexanderdean alexanderdean changed the title "In memory" storage Add in-memory storage option May 5, 2017
@paulboocock paulboocock changed the title Add in-memory storage option Add in-memory queue option Jun 25, 2021
@paulboocock
Copy link
Contributor

Switch to making the Queue in memory, as there is no need to use the "PersistentQueue" concept when building an in memory buffer option. Also much easier to make this thread safe.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants