How to properly manage warnings when loading documents?

[ehenon]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import { DocxLoader } from '@langchain/community/document_loaders/fs/docx';
import { PDFLoader } from '@langchain/community/document_loaders/fs/pdf';
import { PPTXLoader } from '@langchain/community/document_loaders/fs/pptx';
import { DirectoryLoader } from 'langchain/document_loaders/fs/directory';
import { TextLoader } from 'langchain/document_loaders/fs/text';

(async () => {
  const documentsPath = '<PATH_CONTAINING_INPUT_FILES>';
    const loader = new DirectoryLoader(
    documentsPath,
    {
      '.txt': (path) => new TextLoader(path),
      '.pdf': (path) => new PDFLoader(path),
      '.docx': (path) => new DocxLoader(path),
      '.pptx': (path) => new PPTXLoader(path),
    },
  );
  const docs = await loader.load();
})();

Description

When using this method to load input documents with LangChainJS, it is possible that several types of warnings appear in the console, for example :

Warning: TT: undefined function: 32
Warning: TT: undefined function: 21
Warning: Could not find a preferred cmap table.
Warning: TT: invalid function id: 136
Warning: Required "glyf" table is not found -- trying to recover.

After some research, I understood that some of them could be ignored because they were “false positives” linked to underlying libraries allowing, for example, to read PDFs. In the context of a LangChainJS pipeline, how can we manage these possible warnings in a clean way? I suppose we shouldn't ignore them all?

System Info

@langchain/community v0.2.27
@langchain/core v0.2.23
Node v20.14.0
pnpm v9.3.0

[dosubot[bot]]

Hello, @ehenon! I'm here to help you with any bugs, questions, or contributions. Let's tackle this issue together while we wait for a human maintainer to assist us.

To properly manage warnings when loading documents using LangChainJS, you can utilize the UnknownHandling enum provided by the MultiFileLoader class. This enum allows you to specify how to handle unknown file types when loading documents. The options include ignoring the unknown file types, logging a warning, or throwing an error.

Here is an example of how to use the MultiFileLoader with the UnknownHandling enum:

import { MultiFileLoader, UnknownHandling } from "path/to/multi_file_loader";

const multiFileLoader = new MultiFileLoader(
  ["path/to/file1.pdf", "path/to/file2.txt"],
  {
    ".pdf": (path: string) => new PDFLoader(path),
  },
  UnknownHandling.Warn // This will log a warning for unknown file types
);

const docs = await multiFileLoader.load();
console.log({ docs });

In the MultiFileLoader class, the load method handles unknown file types based on the unknown parameter:

public async load(): Promise<Document[]> {
  const documents: Document[] = [];

  for (const filePath of this.filePaths) {
    const fullPath = resolve(filePath);
    const fileStat = await stat(fullPath);
    if (fileStat.isDirectory()) {
      console.warn(`Ignoring directory: ${fullPath}`);
      continue;
    }

    const loaderFactory = this.loaders[extname(fullPath)];
    if (loaderFactory) {
      const loader = loaderFactory(fullPath);
      documents.push(...(await loader.load()));
    } else {
      switch (this.unknown) {
        case UnknownHandling.Ignore:
          break;
        case UnknownHandling.Warn:
          console.warn(`Unknown file type: ${fullPath}`);
          break;
        case UnknownHandling.Error:
          throw new Error(`Unknown file type: ${fullPath}`);
        default:
          throw new Error(`Unknown unknown handling: ${this.unknown}`);
      }
    }
  }

  return documents;
}

By setting the unknown parameter to UnknownHandling.Warn, you can ensure that any unknown file types will trigger a warning message, helping you manage warnings effectively when loading documents ^[1][2].

Additionally, you can control the logging level and potentially suppress or customize warning messages from underlying libraries, such as those used for reading PDFs, through the verbosity option in the DocumentInitParameters:

/**
 * - Controls the logging level; the constants
 * from {@link VerbosityLevel } should be used.
 */
verbosity?: number | undefined;

By setting the verbosity parameter, you can manage the logging level. You can also use the stopAtErrors parameter to reject certain promises when the associated PDF data cannot be successfully parsed:

/**
 * - Reject certain promises, e.g.
 * `getOperatorList`, `getTextContent`, and `RenderTask`, when the associated
 * PDF data cannot be successfully parsed, instead of attempting to recover
 * whatever possible of the data. The default value is `false`.
 */
stopAtErrors?: boolean | undefined;

These options provide flexibility in handling logging and error management when working with PDFs in LangChainJS ^[3][4].

To continue talking to Dosu, mention @dosu.