Vivado synthesis and loop limit for IMEM as ROM

[Sam-Vervaeck]

I am hitting the following error when running a synthesis containing a large IMEM as ROM:

ERROR: [Synth 8-403] loop limit (65538) exceeded [[...]/neorv32/neorv32_package.vhd:1131]

Since I only found one discussion mentioning this briefly in this repo I am making this discussion for future people with the same error.

This is what my compiled program produced in sizes:

Memory utilization:
   text	   data	    bss	    dec	    hex	filename
 404916	     44	    500	 405460	  62fd4	main.elf

This is what I have my Vivado IP Block set to:

And this is the bigger log produced by Vivado synthesis:

INFO: [Synth 8-638] synthesizing module 'neorv32_imem' [[...]/src/neorv32/neorv32_imem.vhd:35]
	Parameter IMEM_SIZE bound to: 524288 - type: integer 
	Parameter IMEM_AS_IROM bound to: 1 - type: bool 
ERROR: [Synth 8-403] loop limit (65538) exceeded [[...]/src/neorv32/neorv32_package.vhd:1131]
ERROR: [Synth 8-421] mismatched array sizes in rhs and lhs of assignment [[...]/src/neorv32/neorv32_imem.vhd:49]
ERROR: [Synth 8-285] failed synthesizing module 'neorv32_imem' [[...]/src/neorv32/neorv32_imem.vhd:35]
ERROR: [Synth 8-285] failed synthesizing module 'neorv32_top' [[...]/src/neorv32/neorv32_top.vhd:236]
ERROR: [Synth 8-285] failed synthesizing module 'neorv32_vivado_ip' [[...]/src/neorv32/neorv32_vivado_ip.vhd:241]
ERROR: [Synth 8-285] failed synthesizing module 'CKKS_neorv32_vivado_ip_0_0' [[...]/ip/CKKS_neorv32_vivado_ip_0_0/synth/CKKS_neorv32_vivado_ip_0_0.vhd:94]

I am running out of BRAM blocks and Vivado is switching to registers maybe? Lets calculate. My Genesys2 for which I am synthesising has 16Mbits of BRAM available, so that is about 16000000 / 36000 ~ 445 BRAM 36K blocks. Next to the NEORV32 IP block, I have one other BRAM block in my block design of 64 bits wide and 131072 deep, so that should take 2*131072/1000 ~ 256 BRAM 36K blocks (you can also read this from the block summary).
I have an accelerator hooked up to the CFS of the NEORV32 that takes 97 BRAM 36K blocks. At this point I am left with 445 - 256 - 97 = 92BRAM 36K blocks.
My IMEM is set to 524288 bytes so that is 131072 32-bit words. That should take 1*131072/1000 ~ 132 BRAM 36K blocks, exceeding the 92 that I have available still. Taking from the other discussion:

Vivado will get stuck when it has exhausted all the BlockRAM resources. Then it will try to use general logic to build further memory. It will fail at some time - but that usually takes quite long.

I believe this is also what is happening to my case. I suppose I will see if I can make my BRAM IP block a bit smaller. I can also maybe try from here to increase the loop limit but that will just increase the time Vivado will take to crash I am afraid.

[Sam-Vervaeck]

I made my BRAM IP block smaller. Now it uses only 192 BRAM 36K blocks. That should leave 445 - 192 - 97 = 156 BRAM 36K blocks for the IMEM which is enough for its need of 132 BRAM 36K blocks.

Yet, synthesis failed again with the same error, so this change was not the solution I hoped it would be. I can try increasing the loop limit as well now and see if that helps. I should also note that the block design passes the verify design checkbox in the diagram editor and I can successfully simulate the whole program loaded into IMEM.

I also noticed these warnings in my synthesis log file:

WARNING: [Synth 8-3936] Found unconnected internal register 'fifo_read_async.fifo_read_async_large.r_pnt_ff_reg' and it is trimmed from '2' to '1' bits. [[...]/src/neorv32/neorv32_fifo.vhd:203]
INFO: [Synth 8-256] done synthesizing module 'neorv32_fifo' (1#1) [[...]/src/neorv32/neorv32_fifo.vhd:43]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[rf_rs1] was removed.  [[...]src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[rf_rs2] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[rf_rd] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[lsu_mo_we] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[lsu_priv] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[ir_funct3] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[ir_funct12] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[ir_opcode] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[cpu_priv] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[cpu_sleep] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[cpu_trap] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element ctrl_reg[cpu_debug] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:390]
WARNING: [Synth 8-6014] Unused sequential element execute_engine_reg[is_ci] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:565]
WARNING: [Synth 8-6014] Unused sequential element csr_reg[mcounteren_cy] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:1555]
WARNING: [Synth 8-6014] Unused sequential element csr_reg[mcounteren_ir] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:1555]
WARNING: [Synth 8-6014] Unused sequential element csr_reg[mcountinhibit] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:1555]
WARNING: [Synth 8-6014] Unused sequential element csr_reg[dcsr_ebreaku] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:1555]
WARNING: [Synth 8-6014] Unused sequential element csr_reg[dpc] was removed.  [/[...]/src/neorv32/neorv32_cpu_control.vhd:1555]
WARNING: [Synth 8-6014] Unused sequential element csr_reg[dscratch0] was removed.  [/[...]/src/neorv32/neorv32_cpu_control.vhd:1555]
WARNING: [Synth 8-6014] Unused sequential element csr_reg[tdata2] was removed.  [/[...]src/neorv32/neorv32_cpu_control.vhd:1555]
WARNING: [Synth 8-6014] Unused sequential element cnt_gen[0].cnt_reg[lo][0] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2055]
WARNING: [Synth 8-6014] Unused sequential element cnt_gen[0].cnt_reg[hi][0] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2055]
WARNING: [Synth 8-6014] Unused sequential element cnt_gen[0].cnt_reg[ovf][0] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2055]
WARNING: [Synth 8-6014] Unused sequential element cnt_gen[1].cnt_reg[lo][1] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2055]
WARNING: [Synth 8-6014] Unused sequential element cnt_gen[1].cnt_reg[hi][1] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2055]
WARNING: [Synth 8-6014] Unused sequential element cnt_gen[1].cnt_reg[ovf][1] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2055]
WARNING: [Synth 8-6014] Unused sequential element cnt_gen[2].cnt_reg[lo][2] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2055]
WARNING: [Synth 8-6014] Unused sequential element cnt_gen[2].cnt_reg[hi][2] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2055]
WARNING: [Synth 8-6014] Unused sequential element cnt_gen[2].cnt_reg[ovf][2] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2055]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][15] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][14] was removed.  [[...]/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][13] was removed.  [[...]/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][12] was removed.  [[...]/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][11] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][10] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][9] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][8] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][7] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][6] was removed.  [...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][5] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][4] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][3] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][2] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][1] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element cnt_reg[inc][0] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:2076]
WARNING: [Synth 8-6014] Unused sequential element csr_reg[tdata1_execute] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:1555]
WARNING: [Synth 8-6014] Unused sequential element csr_reg[tdata1_dmode] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:1555]
WARNING: [Synth 8-6014] Unused sequential element csr_reg[tdata1_action] was removed.  [[...]/src/neorv32/neorv32_cpu_control.vhd:1555]

I am on commit e3136f4b13200750cfa91c39ee826533a069826a aka 🐛 fix minor regression bug; minor RTL optimizations (#998). @stnolting I assume the GitHub Verified tag means synthesis of NEO worked on your side for that commit but you confirm that?

[Sam-Vervaeck]

I got my design synthesised! It only took 5h10min though 😅 (on my older Dell laptop). I increased the loop limit by running this command before synthesis in the tcl console:

set_param synth.elaboration.rodinMoreOptions "rt::set_parameter max_loop_limit 150000"

I chose for 150000 because my IMEM code is about 131K long, so wanted so headroom.
I don't think the need for stuff like this is mentioned anywhere in the datasheet/user guide when the IMEM ROM init is large?

[stnolting]

I am running out of BRAM blocks and Vivado is switching to registers maybe?

Running out of resources is not the problem here (this would cause a different error in a later synthesis step).

Yet, synthesis failed again with the same error, so this change was not the solution I hoped it would be. I can try increasing the loop limit as well now and see if that helps.

I didn't even know that there was an explicit limit for a loop's number of iterations. I always thought the maximum number was only limited by the size of the loop variable (i.e. 32-bit for integers). 🤯

I got my design synthesised! It only took 5h10min though 😅 (on my older Dell laptop). I increased the loop limit by running this command before synthesis in the tcl console:

Nice fix! 👍

I think this extreme runtime is caused by the heavy loop unrolling done by the synthesizer. So this fix is not really feasible... 🙈

My suggestion would be to use an external RAM for the IMEM (a Xilinx IP) and initialize that with a MIF or COE file (note that the NEORV32 software framework provides an option for generating such files: https://stnolting.github.io/neorv32/#_executable_image_generator).

But that's not really nice either... I'll take a closer look at the problematic function:

neorv32/rtl/core/neorv32_package.vhd

Lines 1067 to 1079 in 1d1a173

	-- Initialize mem32_t array from another mem32_t array ------------------------------------
	-- -------------------------------------------------------------------------------------------
	impure function mem32_init_f(init : mem32_t; depth : natural) return mem32_t is
	variable mem_v : mem32_t(0 to depth-1) := (others => (others => '0'));
	begin
	if (init'length > depth) then
	return mem_v;
	end if;
	for i in 0 to init'length-1 loop -- initialize only in range of source data array
	mem_v(i) := init(i);
	end loop;
	return mem_v;
	end function mem32_init_f;

Maybe replacing the for loop by a while loop might help?! Or maybe we do not need a loop here at all...? 🤔

[stnolting]

#1005 hopefully solves this issue.

[Sam-Vervaeck]

I'll run a synthesis tomorrow without the Vivado loop increase to see if it worked :)

[Sam-Vervaeck]

My synthesis was successful this time without having the loop limit variable set (so default loop limit). Seems like the fix works! Great job :)

[stnolting]

I'm delighted to hear that! Thank you very much for the feedback.