Skip to content
This repository has been archived by the owner on Oct 25, 2020. It is now read-only.

panic: didn't find any token of type TokenIdent #7

Closed
jbmcfarlin31 opened this issue Dec 9, 2019 · 2 comments · Fixed by #11
Closed

panic: didn't find any token of type TokenIdent #7

jbmcfarlin31 opened this issue Dec 9, 2019 · 2 comments · Fixed by #11

Comments

@jbmcfarlin31
Copy link

On another repo I tried this tool on, it worked wonderfully! I decided to try it on another branch of one of my other repos and it failed with the following:

panic: didn't find any token of type TokenIdent

goroutine 1 [running]:
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionType(0xc00010cba0, 0x0, 0x0, 0xc0001160f8, 0x0, 0x0, 0x49, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:88 +0x580
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionTypeSingle(0xc00010cba0, 0x0, 0x0, 0xc0001160f8, 0x0, 0x0, 0xc000000049, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:92 +0xe3
github.com/hashicorp/hcl/v2/hclwrite.parseTraversalStep(0x709140, 0xc00009aaa0, 0xc00010cba0, 0x0, 0x0, 0xc0001160f8, 0x0, 0x0, 0xc00010cba0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:395 +0x364
github.com/hashicorp/hcl/v2/hclwrite.parseTraversal(0xc00000f0c0, 0x2, 0x2, 0xc00010cba0, 0x0, 0x0, 0xc0001160f8, 0x0, 0x0, 0xc00010cba0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:377 +0x3da
github.com/hashicorp/hcl/v2/hclwrite.parseExpression(0x70a780, 0xc0000a2160, 0xc00010cba0, 0x0, 0x0, 0xc0001160f8, 0x0, 0x0, 0x8)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:356 +0x175
github.com/hashicorp/hcl/v2/hclwrite.parseAttribute(0xc0000b47e0, 0xc00010cba0, 0x5, 0x5, 0xc0001160f8, 0x5, 0x5, 0xc00010ca80, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:260 +0xaf4
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x707960, 0xc0000b47e0, 0xc00010ca80, 0x46, 0x46, 0xc0001160e0, 0x46, 0x46, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:217 +0x632
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc0000a2370, 0xc00010ca80, 0x46, 0x46, 0xc0001160e0, 0x46, 0x46, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x68d
github.com/hashicorp/hcl/v2/hclwrite.parseBlock(0xc00009e240, 0xc00010ca80, 0x0, 0x0, 0xc0001160e0, 0x0, 0x0, 0xc00010c840, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:329 +0xf72
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x7079a0, 0xc00009e240, 0xc00010c840, 0xd0, 0xd0, 0xc0001160b0, 0xd0, 0xd0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:219 +0x4f7
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc0000a2790, 0xc00010c360, 0xdd, 0xdd, 0xc000116048, 0xdd, 0xdd, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x68d
github.com/hashicorp/hcl/v2/hclwrite.parseBlock(0xc00009ea20, 0xc00010c360, 0x0, 0x0, 0xc000116048, 0x0, 0x0, 0xc00010c000, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:329 +0xf72
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x7079a0, 0xc00009ea20, 0xc00010c000, 0xe8, 0xe8, 0xc000116000, 0xe8, 0xe8, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:219 +0x4f7
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc0000a2840, 0xc00010c000, 0xe9, 0x100, 0xc000116000, 0xe9, 0xe9, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x68d
github.com/hashicorp/hcl/v2/hclwrite.parse(0xc0000f2000, 0x769, 0x969, 0xc0000201c0, 0x34, 0x1, 0x1, 0x0, 0xffffffffffffff9c, 0xc000020200, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:53 +0x2e4
github.com/hashicorp/hcl/v2/hclwrite.ParseConfig(...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/public.go:27
main.processFile(0xc0000201c0, 0x34, 0x1a4)
        /tmp/terraform-clean-syntax/main.go:78 +0xc8
main.processItem(0xc0000201c0, 0x34)
        /tmp/terraform-clean-syntax/main.go:55 +0x2e7
main.processDir(0x7ffcdd7ba68a, 0x2d)
        /tmp/terraform-clean-syntax/main.go:67 +0x119
main.processItem(0x7ffcdd7ba68a, 0x2e)
        /tmp/terraform-clean-syntax/main.go:47 +0x1cc
main.main()
        /tmp/terraform-clean-syntax/main.go:30 +0xec

I re-cloned this repo, rebuilt the binary, and tried again but nothing changed, still the same error.
terraform-clean /my/dir/

Thanks!
@apparentlymart
Copy link
Owner

Hi @jbmcfarlin31!

885338e should've fixed issues of this type, but perhaps you've found a new problem with similar symptoms.

If you rebuild from the latest code on master then you'll get the change from #6 that should allow the program to complete in spite of this crash, and so at least it should be able to update the other files that don't run into this problem. If you're then able to share the source code of the file that is indicated as the cause of the crash, hopefully we can figure out what part of the input is not being handled correctly by the parser.

@jbmcfarlin31
Copy link
Author

jbmcfarlin31 commented Dec 11, 2019 •
edited
Loading

Hi @apparentlymart looks like this is happening with a third party provider. I did a recent pull of this repo and rebuilt the binary. Then I tested in my branch one file at a time until I found the culprit.

The error I receive is the following:

Recovered in processFile while processing rke.tf: "didn't find any token of type TokenIdent"
goroutine 1 [running]:
runtime/debug.Stack(0xc00011b970, 0x624e40, 0xc000124160)
        /root/go/src/runtime/debug/stack.go:24 +0x9d
main.processFile.func1(0xc000014508, 0x6)
        /tmp/terraform-clean-syntax/main.go:82 +0x6e
panic(0x624e40, 0xc000124160)
        /root/go/src/runtime/panic.go:522 +0x1b5
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionType(0xc0000f6ba0, 0x0, 0x0, 0xc0001220f8, 0x0, 0x0, 0x49, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:88 +0x57d
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionTypeSingle(0xc0000f6ba0, 0x0, 0x0, 0xc0001220f8, 0x0, 0x0, 0xc000000049, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:92 +0xe3
github.com/hashicorp/hcl/v2/hclwrite.parseTraversalStep(0x6fa820, 0xc00007c910, 0xc0000f6ba0, 0x0, 0x0, 0xc0001220f8, 0x0, 0x0, 0xc0000f6ba0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:395 +0x365
github.com/hashicorp/hcl/v2/hclwrite.parseTraversal(0xc00000ee60, 0x2, 0x2, 0xc0000f6ba0, 0x0, 0x0, 0xc0001220f8, 0x0, 0x0, 0xc0000f6ba0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:377 +0x3dc
github.com/hashicorp/hcl/v2/hclwrite.parseExpression(0x6fbca0, 0xc0000822c0, 0xc0000f6ba0, 0x0, 0x0, 0xc0001220f8, 0x0, 0x0, 0x8)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:356 +0x178
github.com/hashicorp/hcl/v2/hclwrite.parseAttribute(0xc0000982a0, 0xc0000f6ba0, 0x5, 0x5, 0xc0001220f8, 0x5, 0x5, 0xc0000f6a80, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:260 +0xafe
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x6f9040, 0xc0000982a0, 0xc0000f6a80, 0x46, 0x46, 0xc0001220e0, 0x46, 0x46, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:217 +0x632
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc000082420, 0xc0000f6a80, 0x46, 0x46, 0xc0001220e0, 0x46, 0x46, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x6bb
github.com/hashicorp/hcl/v2/hclwrite.parseBlock(0xc000080240, 0xc0000f6a80, 0x0, 0x0, 0xc0001220e0, 0x0, 0x0, 0xc0000f6840, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:329 +0xf7f
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x6f9080, 0xc000080240, 0xc0000f6840, 0xd0, 0xd0, 0xc0001220b0, 0xd0, 0xd0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:219 +0x4f7
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc000082840, 0xc0000f6360, 0xdd, 0xdd, 0xc000122048, 0xdd, 0xdd, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x6bb
github.com/hashicorp/hcl/v2/hclwrite.parseBlock(0xc000080900, 0xc0000f6360, 0x0, 0x0, 0xc000122048, 0x0, 0x0, 0xc0000f6000, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:329 +0xf7f
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x6f9080, 0xc000080900, 0xc0000f6000, 0xe8, 0xe8, 0xc000122000, 0xe8, 0xe8, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:219 +0x4f7
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc0000828f0, 0xc0000f6000, 0xe9, 0x100, 0xc000122000, 0xe9, 0xe9, 0x0, 0x0, 0x0, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:193 +0x6bb
github.com/hashicorp/hcl/v2/hclwrite.parse(0xc0000c4000, 0x765, 0x965, 0xc000014508, 0x6, 0x1, 0x1, 0x0, 0x490beb, 0x106, ...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/parser.go:53 +0x2e4
github.com/hashicorp/hcl/v2/hclwrite.ParseConfig(...)
        /root/odev/pkg/mod/github.com/hashicorp/hcl/v2@v2.1.1-0.20191206020329-84e71e9393a0/hclwrite/public.go:27
main.processFile(0xc000014508, 0x6, 0x1a4)
        /tmp/terraform-clean-syntax/main.go:86 +0x119
main.processItem(0xc000014508, 0x6)
        /tmp/terraform-clean-syntax/main.go:57 +0x2ea
main.processDir(0x7ffc2fa5368b, 0x1)
        /tmp/terraform-clean-syntax/main.go:69 +0x11a
main.processItem(0x7ffc2fa5368b, 0x1)
        /tmp/terraform-clean-syntax/main.go:49 +0x1cc
main.main()
        /tmp/terraform-clean-syntax/main.go:32 +0xea

The terraform code in question is:

# ---------------------------------------------------------------------
# RKE configuration

resource rke_cluster "cluster" {
  depends_on = [azurestack_public_ip.vmpip, azurestack_virtual_machine.vm]

  dynamic "nodes" {
    for_each = azurestack_public_ip.vmpip.*
    iterator = nodes

    content {
      address = nodes.value.ip_address
      user    = "testuser"
      #role    = nodes.key < var.nbr_managers ? ["controlplane","etcd"] : ["worker"]
      role    = ["controlplane","etcd", "worker"]
      ssh_key = file("/opt/${var.deployment_name}/${var.deployment_name}")
    }
  }

  ignore_docker_version = true
  cluster_name = "${var.deployment_name}-cluster"

  # Kubernetes version
  kubernetes_version = "v1.15.3-rancher1-1"

  private_registries {
    url      = "my.repo.com:1234"
  }

  #########################################################
  # Network(CNI) - supported: flannel/calico/canal/weave
  #########################################################
  # There are several network plug-ins that work, but we default to canal
  network {
    plugin = "weave"
  }

  ingress {
    provider = "nginx"
    extra_args = {
      default-ssl-certificate = "ingress-nginx/wildcard-ingress"
    }
  }

  services_kube_api {
    extra_args = {
      enable-admission-plugins = "ServiceAccount,PodPreset,NamespaceLifecycle,LimitRanger,PersistentVolumeLabel,DefaultStorageClass,ResourceQuota,DefaultTolerationSeconds"
      runtime-config = "batch/v2alpha1,authentication.k8s.io/v1beta1=true,settings.k8s.io/v1alpha1=true"
    }
  }

  services_kubelet {
    extra_args = {
      volume-plugin-dir = "/usr/libexec/kubernetes/kubelet-plugins/volume/exec"
    }
    # Optionally define additional volume binds to a service
    extra_binds = [
      "/usr/libexec/kubernetes/kubelet-plugins/volume/exec:/usr/libexec/kubernetes/kubelet-plugins/volume/exec",
    ]
  }
}

It does look like the latest build of the binary allowed the other files to build despite having a panic. So that's a plus.

This was referenced May 13, 2020
Merged
Merged
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

dependencies: go get github.com/hashicorp/hcl/v2@v2.5.1 alisdair/terraform-clean-syntax
2 participants
@apparentlymart @jbmcfarlin31