On this page

This guide explains how to create new documentation generators for @doc-kit/core.

Generators in doc-kit transform API documentation through a pipeline. Each generator:

  1. Takes input from a previous generator or raw files
  2. Processes the data into a different format
  3. Yields output for the next generator or final output
Raw Markdown Files
    ↓
  [ast] - Parse to MDAST
    ↓
  [metadata] - Extract structured metadata
    ↓
  [jsx-ast] - Convert to JSX AST
    ↓
  [html] - Generate HTML/CSS/JS bundles
text

Each generator declares its dependency using the dependsOn field, allowing automatic pipeline construction. Each stage is documented alongside the rest of the generators.

A generator is defined as a module exporting an object conforming to the GeneratorMetadata interface.

Create a new directory in your project:

/
├── index.mjs         # Generator metadata (required)
├── generate.mjs      # Generator implementation (required)
├── constants.mjs     # Constants (optional)
├── types.d.ts        # TypeScript types (required)
└── utils/            # Utility functions (optional)
    └── formatter.mjs
text

Create a types.d.ts file containing a Generator export. Use this when typing your generator.

export type Generator = GeneratorMetadata<
  {
    // If your generator supports a custom configuration,
    // define it here
    myCustomOption: string;
  },
  Generate<InputToMyGenerator, Promise<OutputOfMyGenerator>>,
  // If your generator supports parallel processing:
  ProcessChunk<
    InputToMyParallelProcessor,
    OutputOfMyParallelProcessor,
    DependenciesOfMyParallelProcessor
  >
>;
TypeScript

A generator module's default export is a plain object with its metadata and implementation. Create it in index.mjs:

import { generate } from './generate.mjs';

/**
 * Generates output in MyFormat.
 *
 * @type {import('./types').Generator}
 */
export default {
  name: 'my-format',

  description: 'Generates documentation in MyFormat',

  // This generator depends on the metadata generator. Dependencies are
  // declared as import specifiers, so they can live in any package.
  dependsOn: '@doc-kit/core/metadata',

  defaultConfiguration: {
    // If your generator supports a custom configuration, define the defaults here
    myCustomOption: 'myDefaultValue',

    // All generators support options in the GlobalConfiguration object
    // To override the defaults, they can be specified here
    ref: 'overriddenRef',
  },

  generate,
};
JavaScript

Create the generator implementation in generate.mjs:

import { writeFile } from 'node:fs/promises';
import { join } from 'node:path';

import getConfig from '../../utils/configuration/index.mjs';

/**
 * Main generation function
 *
 * @type {import('./types').Generator['generate']}
 */
export async function generate(input, worker) {
  const config = getConfig('my-format');

  // Transform input to your format
  const result = transformToMyFormat(input, config.version);

  // Write to file if output directory specified
  if (config.output) {
    await writeFile(
      join(config.output, 'documentation.myformat'),
      result,
      'utf-8'
    );
  }

  return result;
}

/**
 * Transform metadata entries to MyFormat
 * @param {Array<MetadataEntry>} entries
 * @param {import('semver').SemVer} version
 * @returns {string}
 */
function transformToMyFormat(entries, version) {
  // Your transformation logic here
  return entries
    .map(entry => `${entry.api}: ${entry.heading.data.name}`)
    .join('\n');
}
JavaScript

Generators are loaded dynamically by import specifier. Anything that resolves to a module whose default export is a generator works as a --target:

# A package (subpath) export
npx @doc-kit/cli generate -t @my-scope/my-package/my-format ...

# A local file
npx @doc-kit/cli generate -t ./generators/my-format/index.mjs ...
Shell

Built-in generators additionally get a shorthand alias in packages/core/src/generators/index.mjs, which maps the name users type to the import specifier it resolves to:

export const publicGenerators = {
  'json-simple': '@doc-kit/core/json-simple',
  'my-format': '@doc-kit/core/my-format', // Add this
  // ... other generators
};
JavaScript

If the generator lives in this repository, also add a matching subpath to the exports map of its package's package.json.

For generators processing large datasets, implement parallel processing using worker threads.

First, define the generator metadata in index.mjs:

import { generate, processChunk } from './generate.mjs';

/**
 * @type {import('./types').Generator}
 */
export default {
  name: 'parallel-generator',

  description: 'Processes data in parallel',

  dependsOn: '@doc-kit/core/metadata',

  // Indicates this generator has a processChunk implementation
  hasParallelProcessor: true,

  generate,
  processChunk,
};
JavaScript

Then, implement both processChunk and generate in generate.mjs:

import getConfig from '../../utils/configuration/index.mjs';

/**
 * Process a chunk of items in a worker thread.
 * This function runs in isolated worker threads.
 *
 * @type {import('./types').Generator['processChunk']}
 */
export async function processChunk(fullInput, itemIndices, deps) {
  const results = [];

  // Process only the items at specified indices
  for (const idx of itemIndices) {
    const item = fullInput[idx];
    const result = await processItem(item, deps);
    results.push(result);
  }

  return results;
}

/**
 * Main generation function that orchestrates worker threads
 *
 * @type {import('./types').Generator['generate']}
 */
export async function* generate(input, worker) {
  // Configuration for this generator is based on its name
  const config = getConfig('my-format');

  // Prepare serializable dependencies
  const deps = {
    version: config.version,
    // ...other config
  };

  // Stream chunks as they complete
  for await (const chunkResult of worker.stream(input, deps)) {
    // Process chunk result if needed
    yield chunkResult;
  }
}
JavaScript
  1. processChunk executes in worker threads - No access to main thread state
  2. Only serializable data can be passed to workers (no functions, classes, etc.)
  3. fullInput and itemIndices - Workers receive full input but only process specified indices
  4. deps must be serializable - Pass only JSON-compatible data

Use parallel processing when:

  • Processing many independent items (files, modules, entries)
  • Each item takes significant time to process
  • Operations are CPU-intensive

Don't use workers when:

  • Items have dependencies on each other
  • Output must be in specific order
  • Operation is I/O bound rather than CPU bound

Generators can yield results as they're produced using async generators.

Define the generator metadata in index.mjs:

import { generate, processChunk } from './generate.mjs';

/**
 * @type {import('./types').Generator}
 */
export default {
  name: 'streaming-generator',

  description: 'Streams results as they are ready',

  dependsOn: '@doc-kit/core/metadata',

  hasParallelProcessor: true,

  generate,
  processChunk,
};
JavaScript

Implement the generator in generate.mjs:

/**
 * Process a chunk of data
 *
 * @type {import('./types').Generator['processChunk']}
 */
export async function processChunk(fullInput, itemIndices, deps) {
  // Process chunk
  return results;
}

/**
 * Generator function that yields results incrementally
 *
 * @type {import('./types').Generator['generate']}
 */
export async function* generate(input, worker) {
  // Stream results as workers complete chunks
  for await (const chunkResult of worker.stream(input, {})) {
    // Yield immediately - downstream can start processing
    yield chunkResult;
  }
}
JavaScript
  • Reduced memory usage - Process data in chunks
  • Earlier downstream starts - Next generator can begin before this one finishes
  • Better parallelism - Multiple generators can work simultaneously

Some generators must collect all input before processing.

Generator metadata in index.mjs:

import { generate } from './generate.mjs';

/**
 * @type {import('./types').Generator}
 */
export default {
  name: 'batch-generator',

  description: 'Requires all input at once',

  dependsOn: '@doc-kit/generator-react/jsx-ast',

  generate,
};
JavaScript

Implementation in generate.mjs:

/**
 * Non-streaming - returns Promise instead of AsyncGenerator
 *
 * @type {import('./types').Generator['generate']}
 */
export async function generate(input, worker) {
  // Collect all input (if dependency is streaming, this waits for completion)
  const allData = await collectAll(input);

  // Process everything together
  const result = processBatch(allData);

  return result;
}
JavaScript

Use non-streaming when:

  • You need all data to make decisions (e.g., code splitting, global analysis)
  • Output format requires complete dataset
  • Cross-references between items need resolution

In index.mjs:

import { generate } from './generate.mjs';

export default {
  name: 'my-generator',

  // This generator requires the metadata generator's output. The dependency
  // is an import specifier, so it may point at any installed package.
  dependsOn: '@doc-kit/core/metadata',

  // ... other metadata

  generate,
};
JavaScript

In generate.mjs:

export async function generate(input, worker) {
  // input contains the output from 'metadata' generator
}
JavaScript

dependsOn pulls another generator's output in. The inverse, dependent, pushes a generator's output into another generator's pipeline:

export default {
  name: 'section-pages',

  dependsOn: '@doc-kit/core/metadata',

  // Deliver this generator's output through `html`
  dependent: '@doc-kit/generator-react/html',

  generate,
};
JavaScript

The pipeline splices such a generator in front of the first generator on the way to its dependent that consumes the same dependsOn. Here html depends on jsx-ast, which depends on metadata — so jsx-ast is rewired to read from section-pages instead:

ast → metadata → section-pages → jsx-ast → html

Requesting a generator that declares a dependent runs the dependent's whole pipeline, and the run's result is the dependent's output — -t section-pages produces the html site. Several generators may splice in at the same point; they form a chain. A generator whose dependent pipeline never consumes its dependsOn is an error.

dependent also accepts an array. The generator is spliced into every listed pipeline; when requested, it is delivered through the dependents that are already part of the run, or through all of them when none is:

export default {
  name: 'section-pages',
  dependsOn: '@doc-kit/core/metadata',
  dependent: [
    '@doc-kit/generator-react/html',
    '@doc-kit/generator-react/sitemap',
  ],
  generate,
};
JavaScript

With this, -t section-pages builds the site and the sitemap, while -t html -t section-pages builds just the site.

Use a dependent when a generator transforms an intermediate representation (adding, filtering, or rewriting entries) rather than producing a new output format of its own. See the section-pages generator for a worked example.

In generate.mjs:

import { mkdir, writeFile } from 'node:fs/promises';
import { join } from 'node:path';

import getConfig from '../../utils/configuration/index.mjs';

export async function generate(input, worker) {
  const config = getConfig('my-format');

  if (!config.output) {
    // Return data without writing
    return result;
  }

  // Ensure directory exists
  await mkdir(config.output, { recursive: true });

  // Write single file
  await writeFile(join(config.output, 'output.txt'), content, 'utf-8');

  // Write multiple files
  for (const item of items) {
    await writeFile(
      join(config.output, `${item.name}.txt`),
      item.content,
      'utf-8'
    );
  }

  return result;
}
JavaScript
 
import { cp } from 'node:fs/promises';
import { join } from 'node:path';

import getConfig from '../../utils/configuration/index.mjs';

export async function generate(input, worker) {
  const config = getConfig('my-format');

  if (config.output) {
    // Copy asset directory
    await cp(
      new URL('./assets', import.meta.url),
      join(config.output, 'assets'),
      { recursive: true }
    );
  }

  return result;
}
JavaScript
Previous@node-core/doc-kit-legacyNextCreating commands