# Cluster creation- what is the correct shade plugin to resolve substitutions?

**URL:** <https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275>\
**Category:** Akka Cluster\
**Created:** [June 13, 2018, 1:27am UTC](https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275 "2018-06-13T01:27:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![GSoler](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.akka.io/gsoler/32/507_2.png) [@GSoler](https://discuss.akka.io/u/GSoler)\
**Post date:** [June 13, 2018, 1:27am UTC](https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275/1 "2018-06-13T01:27:20Z")

</div>

I am learning by trying to create a very simple cluster. I am running into reference.conf substitution problems at runtime.

Can anyone please show me a pom.xml used with clusters which shows the correct way to use the shade plugin to resolve substitutions?

This is with what I’ve got:

---- My application.conf-

```auto
akka
{
  actor
  {
    provider = "cluster"
  }
  remote
  {
    netty.tcp
    {
      hostname = "127.0.0.1"
      port = 0
    }
    artery
    {
      enabled = on
      canonical.hostname = "127.0.0.1"
      canonical.port = 0
    }
  }

  cluster
  {
    seed-nodes =
        ["akka://ClusterSystem@127.0.0.1:2551",
         "akka://ClusterSystem@127.0.0.1:2552"]

    # auto downing is NOT safe for production deployments.
    # you may want to use it during development, read more about it in the docs.
    auto-down-unreachable-after = 10s
  }
}

Enable metrics extension in akka-cluster-metrics.
akka.extensions=["akka.cluster.metrics.ClusterMetricsExtension"]

```

---- My code-

```auto
File configFile = new File("my-path-to/target/application.conf");
assert configFile.exists();
Config parsedConfig = ConfigFactory.parseFile(configFile);
assert parsedConfig != null;
parsedConfig.resolve();
assert parsedConfig.isResolved();
this.actorSystem = ActorSystem.create("my-actor-system", parsedConfig);  

```

---- This last line causes the following-

```auto
Exception in thread "main" com.typesafe.config.ConfigException$UnresolvedSubstitution: reference.conf @ jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-remote_2.12/2.5.13/akka-remote_2.12-2.5.13.jar!/reference.conf: 879: Could not resolve substitution to a value: ${akka.stream.materializer}
at com.typesafe.config.impl.ConfigReference.resolveSubstitutions(ConfigReference.java:111)
...

```

---- The shade plugin in my pom.xml-

I combed the forums and tried several shade plugins without success. The most recent is the following shade plugin below.

```
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>3.1.1</version>
      <executions>
        <execution>
          <configuration>
            <transformers>
              <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                <resource>reference.conf</resource>
              </transformer>
            </transformers>
            <relocations>
              <relocation>
                <pattern>akka.remote</pattern>
                <shadedPattern>shaded.akka.remote</shadedPattern>
              </relocation>
            </relocations>
          </configuration>
        </execution>
      </executions>
  </plugin>
```

---

<div class="post-metadata">

**Author:** ![raboof](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.akka.io/raboof/32/2913_2.png) [@raboof](https://discuss.akka.io/u/raboof)\
**Post date:** [June 14, 2018, 1:03pm UTC](https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275/2 "2018-06-14T13:03:09Z")

</div>

How are you packaging and starting your application?

Did you see [https://doc.akka.io/docs/akka/current/general/configuration.html#when-using-jarjar-onejar-assembly-or-any-jar-bundler](https://doc.akka.io/docs/akka/current/general/configuration.html#when-using-jarjar-onejar-assembly-or-any-jar-bundler) ?

---

<div class="post-metadata">

**Author:** ![GSoler](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.akka.io/gsoler/32/507_2.png) [@GSoler](https://discuss.akka.io/u/GSoler)\
**Post date:** [June 14, 2018, 6:05pm UTC](https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275/3 "2018-06-14T18:05:15Z")

</div>

Thank you so much, Arnout, for your help.

I am not trying to do anything remotely complicated like packaging the application. I am a newbie and my issue has _got_ to be much simpler than that.

I have backed off a bit to try figure out the basic problem. Now I have simply copied the example distributed in the akka-samples-cluster-java from the cluster usage section of the docs [Akka Cluster Sample with Java](https://doc.akka.io/docs/akka/current/cluster-usage.html) and placed it in my filesystem. Now application.conf and the other .conf files that came with the example are all in the $CLASSPATH and I made no changes to either SimpleClusterApp.java or SimpleClusterListener.java or to any of the .conf files.

Maven compiled it fine. I added the minimum set of jars to the classpath until I got no more ‘class not found’ runtime exceptions. I set the Java system property -Dconfig.trace=loads when launching so I can verify that application.conf is loaded (along with various other reference.conf). But I still get com.typesafe.config.ConfigException$UnresolvedSubstitution.

Config output + stack trace follows: --------------------

```auto
Loading config from a String akka.remote.netty.tcp.port=2551
akka.remote.artery.canonical.port=2551
Loading config from properties {awt.toolkit=sun.lwawt.macosx.LWCToolkit, java.specification.version=9, sun.cpu.isalist=, sun.jnu.encoding=UTF-8, sun.arch.data.model=64, java.vendor.url=http://java.oracle.com/, sun.boot.library.path=/Library/Java/JavaVirtualMachines/jdk-9.0.4.jdk/Contents/Home/lib, sun.java.command=com.gsoler.akkaClusterLab.SimpleClusterApp, jdk.debug=release, java.specification.vendor=Oracle Corporation, java.home=/Library/Java/JavaVirtualMachines/jdk-9.0.4.jdk/Contents/Home, file.separator=/, java.vm.compressedOopsMode=Zero based, line.separator=
, java.specification.name=Java Platform API Specification, java.vm.specification.vendor=Oracle Corporation, sun.management.compiler=HotSpot 64-Bit Tiered Compilers, java.runtime.version=9.0.4+11, user.name=soler, file.encoding=UTF-8, config.trace=loads, java.io.tmpdir=/var/folders/mf/lspn4g2j59ld7l99m1hd109h0000gn/T/, java.version=9.0.4, java.vm.specification.name=Java Virtual Machine Specification, java.awt.printerjob=sun.lwawt.macosx.CPrinterJob, sun.os.patch.level=unknown, java.library.path=/Users/soler/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:., java.vendor=Oracle Corporation, sun.io.unicode.encoding=UnicodeBig, socksNonProxyHosts=local|*.local|169.254/16|*.169.254/16, gopherProxySet=false, file.encoding.pkg=sun.io, java.class.path=/Users/soler/projects/MavenProjects/mavenbasedir/target/classes:/Users/soler/.m2/repository/com/typesafe/config/1.3.2/config-1.3.2.jar:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar:/Users/soler/.m2/repository/org/scala-lang/scala-library/2.12.4/scala-library-2.12.4.jar:/Users/soler/.m2/repository/com/typesafe/akka/akka-protobuf_2.12/2.5.13/akka-protobuf_2.12-2.5.13.jar:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster_2.12/2.5.13/akka-cluster_2.12-2.5.13.jar:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster-tools_2.12/2.5.13/akka-cluster-tools_2.12-2.5.13.jar:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster-metrics_2.12/2.5.13:/Users/soler/.m2/repository/com/typesafe/akka/akka-remote_2.12/2.5.13/akka-remote_2.12-2.5.13.jar, java.vm.vendor=Oracle Corporation, user.timezone=, os.name=Mac OS X, java.vm.specification.version=9, sun.java.launcher=SUN_STANDARD, user.country=US, http.nonProxyHosts=local|*.local|169.254/16|*.169.254/16, sun.cpu.endian=little, user.home=/Users/soler, user.language=en, java.awt.graphicsenv=sun.awt.CGraphicsEnvironment, ftp.nonProxyHosts=local|*.local|169.254/16|*.169.254/16, path.separator=:, os.version=10.13.5, java.runtime.name=Java(TM) SE Runtime Environment, java.vm.name=Java HotSpot(TM) 64-Bit Server VM, java.vendor.url.bug=http://bugreport.java.com/bugreport/, user.dir=/Users/soler/projects/MavenProjects/akka-cluster-lab, os.arch=x86_64, java.vm.info=mixed mode, java.vm.version=9.0.4+11, java.class.version=53.0}
Loading config from resource 'application.conf' URL file:/Users/soler/projects/MavenProjects/akka-cluster-lab/target/classes/application.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: file:/Users/soler/projects/MavenProjects/akka-cluster-lab/target/classes/application.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Loading config from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde but there were no resources called application.json
exception loading application.json: java.io.IOException: resource not found on classpath: application.json
Loading config from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde but there were no resources called application.properties
exception loading application.properties: java.io.IOException: resource not found on classpath: application.properties
Loading config from resource 'reference.conf' URL jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/reference.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/reference.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Looking for 'version.conf' relative to ParseableResourceURL(jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/reference.conf)
Looking for 'version.json' relative to ParseableResourceURL(jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/reference.conf)
Looking for 'version.properties' relative to ParseableResourceURL(jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/reference.conf)
Loading config from resource 'version.conf' URL jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/version.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-actor_2.12/2.5.11/akka-actor_2.12-2.5.11.jar!/version.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Loading config from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde but there were no resources called version.json
exception loading version.json: java.io.IOException: resource not found on classpath: version.json
Loading config from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde but there were no resources called version.properties
exception loading version.properties: java.io.IOException: resource not found on classpath: version.properties
Loading config from resource 'reference.conf' URL jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster_2.12/2.5.13/akka-cluster_2.12-2.5.13.jar!/reference.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster_2.12/2.5.13/akka-cluster_2.12-2.5.13.jar!/reference.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Loading config from resource 'reference.conf' URL jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster-tools_2.12/2.5.13/akka-cluster-tools_2.12-2.5.13.jar!/reference.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-cluster-tools_2.12/2.5.13/akka-cluster-tools_2.12-2.5.13.jar!/reference.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Loading config from resource 'reference.conf' URL jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-remote_2.12/2.5.13/akka-remote_2.12-2.5.13.jar!/reference.conf from class loader jdk.internal.loader.ClassLoaders$AppClassLoader@4f8e5cde
Loading config from a URL: jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-remote_2.12/2.5.13/akka-remote_2.12-2.5.13.jar!/reference.conf
URL sets Content-Type: 'content/unknown'
'content/unknown' isn't a known content type
Exception in thread "main" com.typesafe.config.ConfigException$UnresolvedSubstitution: reference.conf @ jar:file:/Users/soler/.m2/repository/com/typesafe/akka/akka-remote_2.12/2.5.13/akka-remote_2.12-2.5.13.jar!/reference.conf: 879: Could not resolve substitution to a value: ${akka.stream.materializer}
	at com.typesafe.config.impl.ConfigReference.resolveSubstitutions(ConfigReference.java:111)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow(SimpleConfigObject.java:379)
	at com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:312)
	at com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:398)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow(SimpleConfigObject.java:379)
	at com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:312)
	at com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:398)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow(SimpleConfigObject.java:379)
	at com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:312)
	at com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:398)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow(SimpleConfigObject.java:379)
	at com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:312)
	at com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:398)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.SimpleConfigObject$ResolveModifier.modifyChildMayThrow(SimpleConfigObject.java:379)
	at com.typesafe.config.impl.SimpleConfigObject.modifyMayThrow(SimpleConfigObject.java:312)
	at com.typesafe.config.impl.SimpleConfigObject.resolveSubstitutions(SimpleConfigObject.java:398)
	at com.typesafe.config.impl.ResolveContext.realResolve(ResolveContext.java:179)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:142)
	at com.typesafe.config.impl.ResolveContext.resolve(ResolveContext.java:231)
	at com.typesafe.config.impl.SimpleConfig.resolveWith(SimpleConfig.java:78)
	at com.typesafe.config.impl.SimpleConfig.resolve(SimpleConfig.java:68)
	at com.typesafe.config.impl.SimpleConfig.resolve(SimpleConfig.java:63)
	at com.typesafe.config.impl.SimpleConfig.resolve(SimpleConfig.java:41)
	at com.typesafe.config.impl.ConfigImpl$1.call(ConfigImpl.java:371)
	at com.typesafe.config.impl.ConfigImpl$1.call(ConfigImpl.java:364)
	at com.typesafe.config.impl.ConfigImpl$LoaderCache.getOrElseUpdate(ConfigImpl.java:65)
	at com.typesafe.config.impl.ConfigImpl.computeCachedConfig(ConfigImpl.java:92)
	at com.typesafe.config.impl.ConfigImpl.defaultReference(ConfigImpl.java:364)
	at com.typesafe.config.ConfigFactory.defaultReference(ConfigFactory.java:367)
	at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:213)
	at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:182)
	at com.typesafe.config.ConfigFactory$1.call(ConfigFactory.java:259)
	at com.typesafe.config.ConfigFactory$1.call(ConfigFactory.java:256)
	at com.typesafe.config.impl.ConfigImpl$LoaderCache.getOrElseUpdate(ConfigImpl.java:65)
	at com.typesafe.config.impl.ConfigImpl.computeCachedConfig(ConfigImpl.java:92)
	at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:256)
	at com.typesafe.config.ConfigFactory.load(ConfigFactory.java:232)
	at com.gsoler.akkaClusterLab.SimpleClusterApp.startup(SimpleClusterApp.java:24)
	at com.gsoler.akkaClusterLab.SimpleClusterApp.main(SimpleClusterApp.java:13)

```

---

<div class="post-metadata">

**Author:** ![johanandren](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.akka.io/johanandren/32/85_2.png) [@johanandren](https://discuss.akka.io/u/johanandren)\
**Post date:** [June 15, 2018, 8:02am UTC](https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275/4 "2018-06-15T08:02:55Z")

</div>

I’d guess you are still missing the akka-streams artifact on your class-path given that the `akka.stream.materializer` is missing. In general I wouldn’t recommend the “copy artifacts until there is no more class not found exceptions” strategy but rely on the class path the build tool puts together when you have listed all the dependencies.

---

<div class="post-metadata">

**Author:** ![GSoler](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.akka.io/gsoler/32/507_2.png) [@GSoler](https://discuss.akka.io/u/GSoler)\
**Post date:** [June 15, 2018, 6:59pm UTC](https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275/5 "2018-06-15T18:59:07Z")

</div>

You are correct -it was a missing class at runtime!

I would have expected a java.lang.NoClassDefFoundError and didn’t think a missing class would end up being reported as a config UnresolvedSubstitution exception.

BTW - good advice about relying only on the maven compile classpath at runtime.

Thank you Johan for your help!

---

<div class="post-metadata">

**Author:** ![johanandren](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.akka.io/johanandren/32/85_2.png) [@johanandren](https://discuss.akka.io/u/johanandren)\
**Post date:** [June 18, 2018, 11:52am UTC](https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275/6 "2018-06-18T11:52:49Z")

</div>

In this case it is in fact not a missing class, but a missing `reference.conf` from the streams package, which another `reference.conf` in `akka-remote` is expecting to be on the class path (because of Artery, the new remoting implementation being built on top of Akka Streams).

The Typesafe Config library loads all `reference.conf` files from the class paths and then does variable substitution, if a variable is missing it fails like this so that it does not go unnoticed.

---

<div class="post-metadata">

**Author:** ![GSoler](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.akka.io/gsoler/32/507_2.png) [@GSoler](https://discuss.akka.io/u/GSoler)\
**Post date:** [June 18, 2018, 5:58pm UTC](https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275/7 "2018-06-18T17:58:42Z")

</div>

Thank you for the clarification.  
What is the correct course of action then? add a jar with the missing `reference.conf` (as I did) or extract the missing `reference.conf` from the jar and add it by itself to the classpath?

---

<div class="post-metadata">

**Author:** ![johanandren](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.akka.io/johanandren/32/85_2.png) [@johanandren](https://discuss.akka.io/u/johanandren)\
**Post date:** [June 20, 2018, 9:28am UTC](https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275/8 "2018-06-20T09:28:43Z")

</div>

It sounds like a bad gamble to guess that only some parts of a dependency is needed unless you are completely sure about that, so adding the entire jar is the way to go.

---

<div class="post-metadata">

**Author:** ![GSoler](https://sea2.discourse-cdn.com/flex020/user_avatar/discuss.akka.io/gsoler/32/507_2.png) [@GSoler](https://discuss.akka.io/u/GSoler)\
**Post date:** [June 20, 2018, 5:44pm UTC](https://discuss.akka.io/t/cluster-creation-what-is-the-correct-shade-plugin-to-resolve-substitutions/1275/9 "2018-06-20T17:44:08Z")

</div>

Makes sense.  
Thank you all for your help.