📖 Documentation

PackageVersion

Navigation

74 sectionsv0.1

📄 Migrating from Viper to Config Files — glaze help migrating-from-viper-to-config-files

migrating-from-viper-to-config-files

Migrating from Viper to Config Files

Step-by-step guide to migrate existing Glazed applications from Viper-based configuration to the new config file middleware system

Tutorialtutorialmigrationconfigurationmiddlewaresviper

Migrating from Viper to Config Files

Glazed has moved away from Viper-based configuration parsing to a more explicit, traceable config file middleware system. This migration guide walks you through updating existing applications to use the new approach, which provides better observability, deterministic precedence, and cleaner separation between config sources.

The new system replaces Viper's automatic config discovery and merging with explicit file loading middlewares that record each parse step. This makes it clear where each field value originated and enables better debugging with --print-parsed-fields.

Overview of Changes

The migration involves three main areas:

Config File Loading: Replace older Viper-based loading with LoadFieldsFromFile() / LoadFieldsFromFiles() / FromFiles() and env updates
Logging Initialization: Move to InitLoggerFromCobra() or SetupLoggingFromValues()
Cobra Integration: Use CobraParserConfig to wire config discovery, environment variables, and file loading into your commands

⚠️ Critical: Config File Changes Required

Two breaking changes require immediate attention:

1. Config File Discovery No Longer Automatic

Before (Viper): Automatic discovery in standard paths:

viper.AddConfigPath("$HOME/.myapp")
viper.AddConfigPath("/etc/myapp")
viper.ReadInConfig()  // Searches automatically

After: Explicit discovery required:

// Option A: use a declarative config plan directly
plan := config.NewPlan(
    config.WithLayerOrder(config.LayerSystem, config.LayerUser, config.LayerExplicit),
).Add(
    config.SystemAppConfig("myapp"),
    config.XDGAppConfig("myapp"),
    config.HomeAppConfig("myapp"),
    config.ExplicitFile(explicitPath),
)

// Load it either directly...
sources.FromConfigPlan(plan)

// ...or resolve it first if you want to inspect the report/files.
files, report, err := plan.Resolve(context.Background())
_ = report
_ = files

// Option B: plug the plan into CobraParserConfig
cli.WithParserConfig(cli.CobraParserConfig{
    AppName: "myapp", // env prefix only
    ConfigPlanBuilder: func(parsed *values.Values, cmd *cobra.Command, args []string) (*config.Plan, error) {
        return plan, nil
    },
})

Action required: Every application using Viper config discovery must add explicit config file resolution (see Step 4).

2. Config File Format Must Match Section Structure

Before (Viper): Config structure was flexible - Viper read any keys:

# Flat structure that Viper handled
api-key: "secret"
threshold: 42
log-level: "debug"

After: Config must match section names and fields:

# Section names as top-level keys
demo:
  api-key: "secret"
  threshold: 42
logging:
  log-level: "debug"

If your config doesn't match this structure:

Option A: Restructure your config files (simplest)

Group fields under section names
Update field names to match definitions

Option B: Use pattern-based mapping (for legacy configs)

mapper, _ := patternmapper.NewConfigMapper(sections,
    patternmapper.MappingRule{
        Source:          "api-key",  // Flat config
        TargetSection:     "demo",
        TargetField: "api-key",
    },
)
sources.FromFile("config.yaml",
    middlewares.WithConfigMapper(mapper))

Option C: Use custom mapper function (for complex transformations)

mapper := func(raw interface{}) (map[string]map[string]interface{}, error) {
    // Transform your config to section format
    return map[string]map[string]interface{}{
        "demo": {"api-key": raw["api-key"]},
    }, nil
}
sources.FromFile("config.yaml",
    middlewares.WithConfigFileMapper(mapper))

Action required: Audit your config files and either restructure them or add a mapper (see Step 5).

Step 1: Replace Viper Config Middleware

The primary change is replacing Viper-based middleware with explicit config file middlewares. The old approach relied on Viper's automatic config discovery and merging, while the new approach gives you explicit control over which files are loaded and in what order.

Before: Using GatherFlagsFromViper

import (
    "github.com/go-go-golems/glazed/pkg/cmds/middlewares"
    "github.com/go-go-golems/glazed/pkg/cmds/fields"
)

func GetCommandMiddlewares(cmd *cobra.Command) []middlewares.Middleware {
    return []middlewares.Middleware{
        sources.FromCobra(cmd),
        middlewares.GatherFlagsFromViper(
            sources.WithSource("viper"),
        ),
        sources.FromDefaults(),
    }
}

Viper would automatically:

Search standard config paths ($HOME/.app, /etc/app, etc.)
Load config files based on app name
Merge environment variables
Bind command flags

After: Using LoadFieldsFromFile

import (
    "github.com/go-go-golems/glazed/pkg/cmds/middlewares"
    "github.com/go-go-golems/glazed/pkg/cmds/fields"
)

func GetCommandMiddlewares(cmd *cobra.Command) []middlewares.Middleware {
    return []middlewares.Middleware{
        sources.FromCobra(cmd),
        sources.FromEnv("APP"),  // Explicit env prefix
        sources.FromFile("config.yaml",
            middlewares.WithParseOptions(
                sources.WithSource("config"),
            ),
        ),
        sources.FromDefaults(),
    }
}

The new approach:

Requires explicit file paths (no magic discovery)
Separates environment variable handling from file loading
Records each config file as a distinct parse step
Applies files in the order you specify (low → high precedence)

Single Config File

For applications with a single config file, use LoadFieldsFromFile:

sources.FromFile("/etc/myapp/config.yaml",
    middlewares.WithParseOptions(
        sources.WithSource("config"),
    ),
)

The config file must match the default structure (section names as top-level keys):

demo:
  api-key: "secret123"
  threshold: 42

Multiple Config Files (Overlays)

For applications that compose configuration from multiple files, use LoadFieldsFromFiles:

sources.FromFiles([]string{
    "base.yaml",
    "env.yaml", 
    "local.yaml",
}, middlewares.WithParseOptions(
    sources.WithSource("config"),
))

Files are applied in order (low → high precedence), and each file is recorded as a separate parse step. The last file's values win.

Step 2: Replace Custom Viper Instances

If you were using GatherFlagsFromCustomViper to load configuration from specific files or other applications, replace it with explicit file loading middlewares.

Before: Using GatherFlagsFromCustomViper

import (
    "github.com/go-go-golems/glazed/pkg/cmds/middlewares"
)

func GetAdvancedMiddlewares(commandSettings *cli.GlazedCommandSettings) []middlewares.Middleware {
    return []middlewares.Middleware{
        sources.FromCobra(cmd),
        
        // Profile-specific override file
        middlewares.GatherFlagsFromCustomViper(
            middlewares.WithConfigFile(
                fmt.Sprintf("/etc/myapp/%s.yaml", commandSettings.Profile),
            ),
            middlewares.WithParseOptions(
                sources.WithSource("profile-overrides"),
            ),
        ),
        
        // Shared configuration from another app
        middlewares.GatherFlagsFromCustomViper(
            middlewares.WithAppName("shared-config"),
            middlewares.WithParseOptions(
                sources.WithSource("shared"),
            ),
        ),
        
        sources.FromDefaults(),
    }
}

After: Using LoadFieldsFromFiles

import (
    "github.com/go-go-golems/glazed/pkg/cmds/middlewares"
    "github.com/go-go-golems/glazed/pkg/cmds/fields"
)

func GetAdvancedMiddlewares(commandSettings *cli.GlazedCommandSettings) []middlewares.Middleware {
    files := []string{}
    
    // Profile-specific override file
    if commandSettings.Profile != "" {
        files = append(files, fmt.Sprintf("/etc/myapp/%s.yaml", commandSettings.Profile))
    }
    
    // Shared configuration (if you have explicit file paths)
    // Note: Cross-app config sharing now requires explicit file paths
    if sharedConfigPath := os.Getenv("SHARED_CONFIG_PATH"); sharedConfigPath != "" {
        files = append(files, sharedConfigPath)
    }
    
    return []middlewares.Middleware{
        sources.FromCobra(cmd),
        sources.FromFiles(files,
            middlewares.WithParseOptions(
                sources.WithSource("config"),
            ),
        ),
        sources.FromDefaults(),
    }
}

Key differences:

No more automatic discovery based on app names
Explicit file paths required
You control the exact order and sources
Cross-app config sharing requires explicit file paths (no WithAppName)

Step 3: Update Logging Initialization

Logging initialization has been simplified to work directly with Cobra flags instead of requiring Viper binding.

Before: Viper-Based Logging

import (
    "github.com/go-go-golems/glazed/pkg/cmds/logging"
    "github.com/spf13/cobra"
    "github.com/spf13/viper"
)

var rootCmd = &cobra.Command{
    Use: "myapp",
    PersistentPreRun: func(cmd *cobra.Command, args []string) {
        // Initialize logger after Cobra parsed flags and Viper loaded config
        err := logging.InitLoggerFromViper()
        cobra.CheckErr(err)
    },
}

func main() {
    // Add logging section
    err := logging.AddLoggingSectionToRootCommand(rootCmd, "myapp")
    cobra.CheckErr(err)
    
    // Bind flags to Viper before initializing logger
    err = viper.BindPFlags(rootCmd.PersistentFlags())
    cobra.CheckErr(err)
    
    // Initialize logger early
    err = logging.InitLoggerFromViper()
    cobra.CheckErr(err)
    
    _ = rootCmd.Execute()
}

After: Cobra-Based Logging

import (
    "github.com/go-go-golems/glazed/pkg/cmds/logging"
    "github.com/spf13/cobra"
)

var rootCmd = &cobra.Command{
    Use: "myapp",
    PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
        // Initialize logger after Cobra parsed flags
        return logging.InitLoggerFromCobra(cmd)
    },
}

func main() {
    // Add logging flags to root command
    _ = logging.AddLoggingSectionToRootCommand(rootCmd, "myapp")
    
    // ... register commands, help system, etc.
    _ = rootCmd.Execute()
}

Key changes:

No Viper binding required
Single initialization point in PersistentPreRunE
Logging reads directly from Cobra flags
Simpler setup with fewer moving parts

Alternative: Initialize from Parsed Sections

If you're using Glazed's middleware system and want logging to respect config file values, initialize from parsed sections instead:

import (
    "github.com/go-go-golems/glazed/pkg/cmds/logging"
    "github.com/go-go-golems/glazed/pkg/cmds/schema"
)

func runCommand(cmd *cobra.Command, args []string) error {
    // ... setup sections and parse ...
    
    err := sources.Execute(schema_, parsed,
        sources.FromFile("config.yaml"),
        sources.FromEnv("APP"),
        sources.FromCobra(cmd),
    )
    if err != nil {
        return err
    }
    
    // Initialize logging from parsed sections (includes config file values)
    err = logging.SetupLoggingFromValues(parsed)
    if err != nil {
        return err
    }
    
    // ... rest of command logic ...
}

Step 4: Update Cobra Command Setup

For CLI applications, use CobraParserConfig to wire config discovery, environment variables, and file loading. This replaces manual Viper setup and middleware chaining.

Before: Manual Viper Setup

import (
    "github.com/go-go-golems/glazed/pkg/cli"
    "github.com/spf13/viper"
)

func buildCommand() (*cobra.Command, error) {
    // ... create command description ...
    
    cobraCmd := cli.NewCobraCommandFromCommandDescription(desc)
    
    // Manual Viper setup
    viper.SetEnvPrefix("MYAPP")
    viper.AddConfigPath("$HOME/.myapp")
    viper.AddConfigPath("/etc/myapp")
    viper.SetConfigName("config")
    viper.SetConfigType("yaml")
    viper.ReadInConfig()
    viper.BindPFlags(cobraCmd.Flags())
    
    return cobraCmd, nil
}

After: Using CobraParserConfig

import (
    "github.com/go-go-golems/glazed/pkg/cli"
    "github.com/go-go-golems/glazed/pkg/cmds/schema"
    "github.com/go-go-golems/glazed/pkg/cmds/values"
    glazedconfig "github.com/go-go-golems/glazed/pkg/config"
)

func buildCommand() (*cobra.Command, error) {
    // ... create command description ...

    cobraCmd, err := cli.BuildCobraCommandFromCommand(command,
        cli.WithParserConfig(cli.CobraParserConfig{
            AppName: "myapp", // Enables env prefix MYAPP_
            ConfigPlanBuilder: func(parsed *values.Values, cmd *cobra.Command, args []string) (*glazedconfig.Plan, error) {
                cs := &cli.CommandSettings{}
                _ = parsed.DecodeSectionInto(cli.CommandSettingsSlug, cs)

                return glazedconfig.NewPlan(
                    glazedconfig.WithLayerOrder(
                        glazedconfig.LayerSystem,
                        glazedconfig.LayerUser,
                        glazedconfig.LayerExplicit,
                    ),
                ).Add(
                    glazedconfig.SystemAppConfig("myapp").Named("system-app-config"),
                    glazedconfig.XDGAppConfig("myapp").Named("xdg-app-config"),
                    glazedconfig.HomeAppConfig("myapp").Named("home-app-config"),
                    glazedconfig.ExplicitFile(cs.ConfigFile).Named("explicit-config"),
                ), nil
            },
        }),
    )

    return cobraCmd, err
}

Benefits:

AppName automatically enables environment variable prefix (MYAPP_)
ConfigPlanBuilder keeps config discovery explicit and provenance-aware
--config-file flag is still available via the command-settings section when your plan wants to consume it
Config files integrate cleanly with env vars and flags

Step 5: Handle Custom Config Structures

If your config files don't match the default section structure, you have two options: pattern-based mapping (declarative) or custom mapper functions (programmatic).

Pattern-Based Mapping (Recommended)

Use pattern-based mapping when you can describe your config structure with patterns. This keeps mapping logic declarative and testable:

import (
    pm "github.com/go-go-golems/glazed/pkg/cmds/middlewares/patternmapper"
)

// Define mapping rules
mapper, err := pm.NewConfigMapper(schema_,
    pm.MappingRule{
        Source:          "app.settings.api_key",
        TargetSection:     "demo",
        TargetField: "api-key",
    },
    pm.MappingRule{
        Source:          "app.{env}.api_key",
        TargetSection:     "demo",
        TargetField: "{env}-api-key",
    },
)

// Use with LoadFieldsFromFile
middleware := sources.FromFile(
    "config.yaml",
    middlewares.WithConfigMapper(mapper),
)

When to use:

Config structure is predictable and can be described with patterns
You want declarative, testable mapping rules
Multiple environments or tenants share similar structures

Custom Mapper Functions

Use custom mapper functions when you need full control over config transformation:

mapper := func(rawConfig interface{}) (map[string]map[string]interface{}, error) {
    configMap := rawConfig.(map[string]interface{})
    result := map[string]map[string]interface{}{
        "demo": make(map[string]interface{}),
    }
    
    // Transform config structure to section format
    if apiKey, ok := configMap["api_key"]; ok {
        result["demo"]["api-key"] = apiKey
    }
    
    // Handle nested structures, arrays, validation, etc.
    // ...
    
    return result, nil
}

middleware := sources.FromFile(
    "config.yaml",
    middlewares.WithConfigFileMapper(mapper),
)

When to use:

Complex transformations that patterns can't express
Conditional logic based on config values
Cross-field validation or derivation
Integration with external config formats

Step 6: Update Middleware Execution Order

The precedence order remains the same, but the way you express it changes. Remember: middlewares execute in reverse order (last middleware runs first).

Correct Precedence Order

sources.Execute(schema_, parsed,
    sources.FromDefaults(),                              // Lowest priority
    sources.FromFiles([]string{               // Config files (low → high)
        "base.yaml",
        "env.yaml", 
        "local.yaml",
    }),
    sources.FromEnv("APP"),                           // Environment variables
    sources.FromArgs(args),                          // Positional arguments
    sources.FromCobra(cmd),                     // Flags (highest priority)
)

Precedence: Defaults < Config Files (low→high) < Env < Args < Flags

Each source overrides the previous ones, and each config file in the list overrides earlier files.

Step 7: Remove Viper Dependencies

After migrating, you can remove Viper-related code:

Remove Viper imports:

// Remove these
import "github.com/spf13/viper"

Remove Viper initialization:

// Remove calls like:
viper.SetEnvPrefix("APP")
viper.AddConfigPath("...")
viper.ReadInConfig()
viper.BindPFlags(...)

Remove deprecated middleware calls:

// Remove:
middlewares.GatherFlagsFromViper(...)
middlewares.GatherFlagsFromCustomViper(...)

Update logging initialization:

// Replace:
logging.InitLoggerFromViper()

// With:
logging.InitLoggerFromCobra(cmd)
// or
logging.SetupLoggingFromValues(parsed)

Common Migration Patterns

Pattern 1: Single Config File with Discovery

Before:

viper.SetConfigName("config")
viper.AddConfigPath("$HOME/.myapp")
viper.AddConfigPath("/etc/myapp")
viper.ReadInConfig()

After:

import glazedconfig "github.com/go-go-golems/glazed/pkg/config"

plan := glazedconfig.NewPlan(
    glazedconfig.WithLayerOrder(
        glazedconfig.LayerSystem,
        glazedconfig.LayerUser,
    ),
).Add(
    glazedconfig.SystemAppConfig("myapp"),
    glazedconfig.XDGAppConfig("myapp"),
    glazedconfig.HomeAppConfig("myapp"),
)

files, _, err := plan.Resolve(context.Background())
if err == nil {
    sources.FromResolvedFiles(files)
}

Pattern 2: Profile-Based Config Files

Before:

middlewares.GatherFlagsFromCustomViper(
    middlewares.WithConfigFile(
        fmt.Sprintf("/etc/myapp/%s.yaml", profile),
    ),
)

After:

files := []string{}
if profile != "" {
    files = append(files, fmt.Sprintf("/etc/myapp/%s.yaml", profile))
}
sources.FromFiles(files)

Pattern 3: Environment Variable Overrides

Before: Viper automatically merged environment variables based on prefix and key naming.

After:

// Explicit env prefix
sources.FromEnv("APP")  // Reads APP_* variables

Environment variable names follow the pattern: {PREFIX}_{SECTION}_{FIELD} (e.g., APP_DEMO_API_KEY for demo.api-key).

Pattern 4: Config File Override Pattern

Before: Manual Viper config file merging with custom precedence.

After:

resolver := func(parsed *values.Values, _ *cobra.Command, _ []string) ([]string, error) {
    cs := &cli.CommandSettings{}
    _ = parsed.DecodeSectionInto(cli.CommandSettingsSlug, cs)
    files := []string{}
    
    if cs.ConfigFile != "" {
        files = append(files, cs.ConfigFile)
        
        // Add override file if it exists
        dir := filepath.Dir(cs.ConfigFile)
        base := filepath.Base(cs.ConfigFile)
        stem := strings.TrimSuffix(base, filepath.Ext(base))
        override := filepath.Join(dir, fmt.Sprintf("%s.override.yaml", stem))
        if _, err := os.Stat(override); err == nil {
            files = append(files, override)
        }
    }
    
    return files, nil
}

Debugging and Validation

Inspect Parse Steps

Use --print-parsed-fields to see exactly where each field value came from:

myapp command --print-parsed-fields

This shows the full parse history for each field, including which config file set each value.

Validate Config Files

Before applying config files, validate them against your section definitions:

import (
    "os"
    "gopkg.in/yaml.v3"
)

func validateConfigFile(schema_ *schema.Schema, path string) error {
    b, err := os.ReadFile(path)
    if err != nil {
        return err
    }
    
    var raw map[string]interface{}
    if err := yaml.Unmarshal(b, &raw); err != nil {
        return err
    }
    
    // Check each section and field
    for sectionSlug, v := range raw {
        section, ok := schema_.Get(sectionSlug)
        if !ok {
            return fmt.Errorf("unknown section: %s", sectionSlug)
        }
        
        kv, ok := v.(map[string]interface{})
        if !ok {
            return fmt.Errorf("section %s must be an object", sectionSlug)
        }
        
        pds := section.GetDefinitions()
        for key, val := range kv {
            pd, ok := pds.Get(key)
            if !ok {
                return fmt.Errorf("unknown field %s.%s", sectionSlug, key)
            }
            
            if _, err := pd.CheckValueValidity(val); err != nil {
                return fmt.Errorf("invalid value for %s.%s: %v", sectionSlug, key, err)
            }
        }
    }
    
    return nil
}

Troubleshooting

Config File Not Found

If your config file isn't being loaded, check:

File path is correct and file exists
Your ConfigPlanBuilder resolves the expected files
File has correct permissions

Environment Variables Not Working

If environment variables aren't being read:

Check the prefix matches your AppName (e.g., MYAPP_ for AppName: "myapp")
Variable names follow {PREFIX}_{SECTION}_{FIELD} format
UpdateFromEnv middleware is included in your middleware chain

Precedence Issues

If values aren't overriding as expected:

Verify middleware order (last middleware runs first)
Check config file order in LoadFieldsFromFiles (low → high)
Use --print-parsed-fields to see actual precedence

Legacy Config Format

If you have legacy config files that don't match the section structure:

Use pattern-based mapping for structured transformations
Use custom mapper functions for complex transformations
Consider migrating config files to the new format over time

Complete Example

Here's a complete example showing a before and after migration:

Before

package main

import (
    "github.com/go-go-golems/glazed/pkg/cli"
    "github.com/go-go-golems/glazed/pkg/cmds/logging"
    "github.com/go-go-golems/glazed/pkg/cmds/middlewares"
    "github.com/spf13/cobra"
    "github.com/spf13/viper"
)

var rootCmd = &cobra.Command{
    Use: "myapp",
    PersistentPreRun: func(cmd *cobra.Command, args []string) {
        err := logging.InitLoggerFromViper()
        cobra.CheckErr(err)
    },
}

func main() {
    err := logging.AddLoggingSectionToRootCommand(rootCmd, "myapp")
    cobra.CheckErr(err)
    
    viper.SetEnvPrefix("MYAPP")
    viper.AddConfigPath("$HOME/.myapp")
    viper.AddConfigPath("/etc/myapp")
    viper.SetConfigName("config")
    viper.SetConfigType("yaml")
    viper.ReadInConfig()
    viper.BindPFlags(rootCmd.PersistentFlags())
    
    err = logging.InitLoggerFromViper()
    cobra.CheckErr(err)
    
    _ = rootCmd.Execute()
}

func GetMiddlewares(cmd *cobra.Command) []middlewares.Middleware {
    return []middlewares.Middleware{
        sources.FromCobra(cmd),
        middlewares.GatherFlagsFromViper(),
        sources.FromDefaults(),
    }
}

After

package main

import (
    "github.com/go-go-golems/glazed/pkg/cli"
    "github.com/go-go-golems/glazed/pkg/cmds/logging"
    appconfig "github.com/go-go-golems/glazed/pkg/config"
    "github.com/spf13/cobra"
)

var rootCmd = &cobra.Command{
    Use: "myapp",
    PersistentPreRunE: func(cmd *cobra.Command, args []string) error {
        return logging.InitLoggerFromCobra(cmd)
    },
}

func main() {
    _ = logging.AddLoggingSectionToRootCommand(rootCmd, "myapp")
    
    // ... register commands with BuildCobraCommandFromCommand ...
    
    _ = rootCmd.Execute()
}

func buildCommand() (*cobra.Command, error) {
    // ... create command description ...
    
    return cli.BuildCobraCommandFromCommand(command,
        cli.WithParserConfig(cli.CobraParserConfig{
            AppName: "myapp",
            ConfigPlanBuilder: func(parsed *values.Values, cmd *cobra.Command, args []string) (*glazedconfig.Plan, error) {
                cs := &cli.CommandSettings{}
                _ = parsed.DecodeSectionInto(cli.CommandSettingsSlug, cs)
                return glazedconfig.NewPlan(
                    glazedconfig.WithLayerOrder(
                        glazedconfig.LayerSystem,
                        glazedconfig.LayerUser,
                        glazedconfig.LayerExplicit,
                    ),
                ).Add(
                    glazedconfig.SystemAppConfig("myapp"),
                    glazedconfig.XDGAppConfig("myapp"),
                    glazedconfig.HomeAppConfig("myapp"),
                    glazedconfig.ExplicitFile(cs.ConfigFile),
                ), nil
            },
        }),
    )
}

Next Steps

After completing the migration:

Test thoroughly: Verify all config sources work correctly
Use --print-parsed-fields: Confirm precedence is as expected
Update documentation: Document your config file locations and formats
Remove Viper: Clean up any remaining Viper dependencies
Consider validation: Add config file validation to catch errors early

For more details on the new config system, see:

glaze help config-files - Config files and overlays guide
glaze help pattern-based-config-mapping - Pattern-based mapping guide
glaze help cmds-middlewares - Middleware system reference