Pelican data processing with jinja2content

Introduction

Continuing our investigations from our previous article about data processing in Pelican themes, we'll now explore another powerful plugin: jinja2content. This plugin extends Pelican's capabilities by allowing us to use Jinja2 code directly within our content files.

With this plugin, we can implement dynamic content generation right in our project's Markdown files. For example:

<ul>
{% for href, caption in [('index.html', 'Index'), ('about.html', 'About'),
                        ('downloads.html', 'Downloads')] %}
    <li><a href="{{ href }}">{{ caption }}</a></li>
{% endfor %}
</ul>

When rendered in the browser, this code produces a neatly formatted list of links. Without the plugin, this code would be displayed literally on the page rather than executed.

In this article, we'll explore:

• What data structures we can use in Pelican's content files
• Whether these data structures will be available for processing by the theme
• How to make this data available for theme processing

How jinja2content works

Before diving into implementation details, it's important to understand how this plugin operates behind the scenes.

According to the plugin's documentation:

This plugin allows the use of Jinja2 directives inside your Pelican articles and pages.

In this approach, your content is first rendered by the Jinja template engine. The result is then passed to the normal Pelican reader as usual. There are two consequences for usage. First, this means the Pelican context and Jinja variables usually visible to your article or page template are not available at rendering time. Second, it means that if any of your input content could be parsed as Jinja directives, they will be rendered as such. This is unlikely to happen accidentally, but it's good to be aware of.

This processing order has significant implications for how we structure our data, as we'll see next.

Compatibility issues with YAML frontmatter

The YAML frontmatter problem

Almost immediately upon testing more complex data structures, we encountered compatibility issues. When this plugin is activated, complex YAML frontmatter structures become inaccessible in our templates.

This occurs because the plugin parses content before passing it to Pelican's parser. Since the content (including YAML frontmatter) is processed by Jinja2 first, structured YAML data can be mangled before Pelican has a chance to process it properly.

The compatibility challenge

If the content is pre-processed by the plugin's reader before reaching Pelican's native parser, we need an alternative approach. Ideally, we would split the data before the initial parsing and reconnect it for later processing by Pelican, but this isn't supported by the plugin architecture.

We need to find another way to represent structured data that works with this processing pipeline.

Data structures in Pelican content

Since complex YAML frontmatter isn't compatible with this plugin, we need alternative ways to define structured data within our content files.

Original YAML approach (incompatible)

For reference, here's how we defined testimonial data in our previous article:

---
title: Testimonial
data:
    testimonials:
        - name: John
          position: Client
          blurb: Will buy again.
        - name: Jill
          position: Business owner
          blurb: Great service!
---

Using nested lists

One approach is to use Jinja2's variable declaration syntax with nested lists:

{% set testimonialsA = ['John', 'Client', 'Will buy again.'] %}
{% set testimonialsB = ['Jill', 'Business owner', 'Great service!'] %}
{% set testimonialsList = [testimonialsA, testimonialsB] %}

Using nested dictionaries

For more semantic clarity, we can use nested dictionaries:

{% set testimonialsDict = {
'testimonial01': {'name': 'John', 'position': 'Client', 'blurb': 'Will buy again.'},
'testimonial02': {'name': 'Jill', 'position': 'Business owner', 'blurb': 'Great service!'},
} %}

This approach preserves the semantic structure while remaining compatible with the jinja2content processing pipeline.

Processing data with theme templates

Consistent with our previous article, our goal remains to keep data processing within the theme's templates. This separation of concerns helps maintain a clean architecture and makes our themes more reusable.

Let's explore three different approaches to processing data defined in our content files.

Including template files

In this approach, we include templates from our theme directly within our content pages.

For example, to display our testimonials, we could add this to our Markdown file:

{% include 'partials/sections/about.html' %}

Then, in our theme's partials/sections/about.html file, we can access and display the data:

<div class="testimonials">
    {% for testimonial in testimonialsList %}
        <div class="testimonial">
            <p class="name">{{ testimonial[0] }}</p>
            <p class="position">{{ testimonial[1] }}</p>
            <p class="blurb">{{ testimonial[2] }}</p>
        </div>
    {% endfor %}
</div>

This works because the data variables are defined before the include statement in our content file.

Importing macros

For more sophisticated processing, we can import macros from our theme and apply them to our data.

In our content file (content/pages/about.md), we define the data and import the macro:

{% set testimonialsDict = {
'testimonial01': {'name': 'John', 'position': 'Client', 'blurb': 'Will buy again.'},
'testimonial02': {'name': 'Jill', 'position': 'Business owner', 'blurb': 'Great service!'},
} %}

{% from 'macros.html' import process_testimonials %}
{{ process_testimonials(testimonialsDict) }}

In our theme's templates/macros.html file, we define the processing macro:

{% macro process_testimonials(testimonialsDict) -%}
    <dl class="testimonials">
        {% for item, dict in testimonialsDict.items() %}
            {% for key, value in dict.items() %}
                <dt>
                    {{ key|e }}
                </dt>
                <dd>
                    {{ value|e }}
                </dd>
            {% endfor %}
        {% endfor %}
    </dl>
{%- endmacro %}

When the site is generated, the macro processes our data into well-formed HTML.

Using inline Jinja

Although it goes against our aim to separate data from presentation, the plugin also allows inline Jinja processing directly in content files:

{% set testimonialsDict = {
'testimonial01': {'name': 'John', 'position': 'Client', 'blurb': 'Will buy again.'},
'testimonial02': {'name': 'Jill', 'position': 'Business owner', 'blurb': 'Great service!'},
} %}

<dl class="testimonials">
    {% for item, dict in testimonialsDict.items() %}
        {% for key, value in dict.items() %}
            <dt>
                {{ key|e }}
            </dt>
            <dd>
                {{ value|e }}
            </dd>
        {% endfor %}
    {% endfor %}
</dl>

This approach might be useful for quick prototyping or simpler sites but generally should be avoided for maintainability.

Best practices

Based on our experiments, here are some recommendations for using jinja2content effectively:

Conclusion

The jinja2content plugin offers powerful capabilities for dynamic content generation in Pelican sites, but comes with some important limitations to consider:

• Complex YAML frontmatter becomes inaccessible when using this plugin
• Data structures must be defined within the content body using Jinja2 syntax
• Several approaches exist for processing this data: including templates, importing macros, or using inline Jinja

By understanding these constraints and leveraging the alternative approaches we've explored, you can effectively use complex data structures in your Pelican projects while maintaining a clean separation between data and presentation.

For more advanced use cases, consider combining this approach with custom plugins that might offer more flexibility in how data is accessed and processed during the site generation pipeline.