How to get all logs through the /loki/api/v1/tail interface

[xiaozhi747]

Describe the bug
Through the documentation, it is found that the /loki/api/v1/tail interface has a limit parameter to set the maximum number of entries to be returned, but I set 200,000, and the number of logs that can be seen is only about 2,000.

To Reproduce
My request connection is as follows: ws://xxx.com/logger/push/loki/api/v1/tail?query={record_step_uuid="74c134467d1b43eebe57467f98aaf01f"}&start=1633676882285000000&limit=200000

Expected behavior
Able to display the number of logs around 200,000

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

Screenshots, Promtail config, or terminal output
Only able to display logs of about 2000 lines

[guobei2028]

I got the same problem.

[guobei2028]

@sandeepsukhani Can you help us for this problem? thank you so much

[jeschkies]

Is there a particular reason to use tail? In any case, you could run a new tail request and set start to the timestamp of the last entry plus one. Say you received an entry at 1642177577802175688 then your next request would be /loki/api/v1/tail?start=1642177577802175689. This relates to #4793.

[guobei2028]

@jeschkies
why need run a new tail request？
For example, when the number of log entries under the current label is 3000, and the limit of the tail interface is 5000,
I should be able to find all the logs with /loki/api/v1/tail?start=1642177577802175688. Should not initiate a new request with entry at 1642177577802175689.

[jeschkies]

Ah, you mean the request didn't return all entries even though it was not exceeding the limit? This may be a silly question but how did you verify the actual number of entries? Did you use GET /loki/api/v1/query?

I'm trying to figure out whether the endpoint has a bug.

[guobei2028]

@jeschkies
yes, when I use GET /loki/api/v1/query_range can get all logs, but GET /loki/api/v1/tail not.
when I use tail, set limit to 5000, and the number of logs that can be seen is only about 2,000.

[slim-bean]

The tail endpoint is not built with the intention of providing all logs for a stream, the primary use case was for a person to watch a stream of logs in near real-time.

The lookback is a rather simplistic implementation to mimic the behavior of tail -n 100 for slowly updating streams to both add context and some indication that there are existing logs.

I would also comment that there are no guarantees with the current implementation that logs can't be missed, if you try to execute a very large query, you can very likely miss log lines between when that query returns and the live tailing starts.

The tail history is just a query that executes with a normal from and to where to is the current time when the query is sent, so if it takes 30s to execute that query you likely will be missing 30s of logs before the live tailing starts.

None of Loki's API's are currently built for returning very large results, the most reliable way to do this would be to make repeated calls to the query_range endpoint (which is what logcli does) to work within the batch limits.

Loki's API's and primary use case is for human interaction, we do discuss machine to machine API's and interfaces regularly and I know this will be added, but in the meantime a form of polling the query_range endpoint is going to be the most reliable way to get logs.

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[vemulavarun]

give an example of loki/api/v1/tail