hdfs backend

[kukougu]

I want to use the ior and mdtest for hdfs, and I use ./configure --with-hdfs and then make. However, I get some error:

aiori-HDFS.c: In function ‘HDFS_Create_Or_Open’:
aiori-HDFS.c:237:12: error: ‘IOR_param_t’ has no member named ‘openFlags’
  if ( param->openFlags & IOR_RDWR ) {
            ^
aiori-HDFS.c:241:12: error: ‘IOR_param_t’ has no member named ‘openFlags’
  if ( param->openFlags & IOR_EXCL ) {
            ^
aiori-HDFS.c:245:12: error: ‘IOR_param_t’ has no member named ‘openFlags’
  if ( param->openFlags & IOR_APPEND ) {
            ^
aiori-HDFS.c:257:12: error: ‘IOR_param_t’ has no member named ‘openFlags’
  if ( param->openFlags & IOR_WRONLY ) {
            ^
aiori-HDFS.c:261:9: error: ‘rank’ undeclared (first use in this function)
    if ( rank != 0 ) {
         ^
aiori-HDFS.c:261:9: note: each undeclared identifier is reported only once for each function it appears in
aiori-HDFS.c:282:12: error: ‘IOR_param_t’ has no member named ‘useO_DIRECT’
  if ( param->useO_DIRECT == TRUE ) {
            ^
aiori-HDFS.c:293:13: error: ‘IOR_param_t’ has no member named ‘openFlags’
  if (( param->openFlags & IOR_WRONLY ) &&
             ^
In file included from aiori.h:22:0,
                 from ior.h:40,
                 from aiori-HDFS.c:85:
aiori-HDFS.c:297:25: error: ‘testComm’ undeclared (first use in this function)
   MPI_CHECK(MPI_Barrier(testComm), "barrier error");
                         ^
aiori-debug.h:97:29: note: in definition of macro ‘MPI_CHECKF’
     int checkf_mpi_status = MPI_STATUS;                                  \
                             ^
aiori-HDFS.c:297:3: note: in expansion of macro ‘MPI_CHECK’
   MPI_CHECK(MPI_Barrier(testComm), "barrier error");
   ^
aiori-HDFS.c:327:13: error: ‘IOR_param_t’ has no member named ‘openFlags’
  if (( param->openFlags & IOR_WRONLY ) &&
             ^
aiori-HDFS.c: In function ‘HDFS_Open’:
aiori-HDFS.c:363:12: error: ‘IOR_param_t’ has no member named ‘openFlags’
  if ( param->openFlags & IOR_CREAT ) {
            ^
aiori-HDFS.c: In function ‘HDFS_Xfer’:
aiori-HDFS.c:392:26: error: ‘IOR_param_t’ has no member named ‘offset’
  off_t     offset = param->offset;
                          ^
aiori-HDFS.c:403:7: error: ‘rank’ undeclared (first use in this function)
       rank,
       ^
aiori-HDFS.c:404:12: error: ‘IOR_param_t’ has no member named ‘offset’
       param->offset + length - remaining);
            ^
aiori-HDFS.c:426:12: error: ‘IOR_param_t’ has no member named ‘offset’
       param->offset + length - remaining );
            ^
aiori-HDFS.c:452:11: error: ‘IOR_param_t’ has no member named ‘offset’
      param->offset + length - remaining );
           ^
aiori-HDFS.c: In function ‘HDFS_Close’:
aiori-HDFS.c:529:12: error: ‘IOR_param_t’ has no member named ‘openFlags’
  if ( param->openFlags & IOR_WRONLY ) {
            ^
aiori-HDFS.c: In function ‘HDFS_Delete’:
aiori-HDFS.c:566:11: error: ‘rank’ undeclared (first use in this function)
           rank, testFileName);
           ^
aiori-HDFS.c: In function ‘HDFS_GetFileSize’:
aiori-HDFS.c:653:9: error: ‘rank’ undeclared (first use in this function)
    if ( rank == 0 ) {
         ^
make[3]: *** [libaiori_a-aiori-HDFS.o] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1

If I do something wrong? What I can do?

[JulianKunkel]

Hi, you hadn't done anything wrong. Indeed the module hasn't been used/tested for quite a long time. Could you give the fix-hdfs referred by #302 a shot to see if that works for you? Note that the arguments for the HDFS module now need to be specified using --hdfs.XXXX, with --help you'll get all the settable options.

[kukougu]

Thanks. It can make now, but It doesn't work. I run ./mdtest -a=HDFS -n=100 -F -C -d /mdtest --hdfs.user=root --hdfs.name_node=namenode --hdfs.replicas=1 --hdfs.block_size=67108864 But there are some bug in option.c. I get these error:

[datanode]#./mdtest -a=HDFS -n=100 -F -C -d /mdtest --hdfs.user=root --hdfs.name_node=namenode --hdfs.replicas=1 --hdfs.block_size=67108864
mdtest: option.c:287: option_parse_token: Assertion `o->type == 'd'' failed.
[datanode:80374] *** Process received signal ***
[datanode:80374] Signal: Aborted (6)
[datanode:80374] Signal code:  (-6)
[datanode:80374] [ 0] /lib64/libpthread.so.0(+0xf630)[0x7fc446c83630]
[datanode:80374] [ 1] /lib64/libc.so.6(gsignal+0x37)[0x7fc4468dc387]
[datanode:80374] [ 2] /lib64/libc.so.6(abort+0x148)[0x7fc4468dda78]
[datanode:80374] [ 3] /lib64/libc.so.6(+0x2f1a6)[0x7fc4468d51a6]
[datanode:80374] [ 4] /lib64/libc.so.6(+0x2f252)[0x7fc4468d5252]
[datanode:80374] [ 5] ./mdtest[0x41c7f7]
[datanode:80374] [ 6] ./mdtest[0x41ca1b]
[datanode:80374] [ 7] ./mdtest[0x41404d]
[datanode:80374] [ 8] ./mdtest[0x403742]
[datanode:80374] [ 9] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fc4468c8555]
[datanode:80374] [10] ./mdtest[0x403777]
[datanode:80374] *** End of error message ***
Aborted (core dumped)

[JulianKunkel]

I see. I'll do some testing. Note: MDTest wasn't supported by the HDFS API so far.

[JulianKunkel]

It appears, it works with IOR now, example in local mode: $ mpiexec -np 2 ./src/ior -F -a HDFS --hdfs.user=root --hdfs.replicas=1 -G 141 -w -r -o /tmp/testfile -t 1m -k -R The error: readDirect: FSDataInputStream#read error: UnsupportedOperationException: Byte-buffer read unsupported by input streamjava.lang.UnsupportedOperationException: Byte-buffer read unsupported by input stream at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:150) Can apparently be ignored. https://issues.apache.org/jira/browse/HDFS-14083 Can you verify?

[JulianKunkel]

I've now implemented the necessary AIORI interface for MDTest. Let me know if there is any problem. Example: $ mpiexec -np 2 ./src/mdtest -a HDFS --hdfs.user=root --hdfs.replicas=1 -n 10 Operation Max Min Mean Std Dev --------- --- --- ---- ------- Directory creation : 22,523 22,523 22,523 0,000 Directory stat : 325,632 325,628 325,630 0,002 Directory removal : 1593,567 1593,198 1593,382 0,184 File creation : 184,086 184,077 184,081 0,004 File stat : 520,211 520,209 520,210 0,001 File read : 1446,575 1446,525 1446,550 0,025 File removal : 5043,560 5043,509 5043,535 0,025 Tree creation : 95,137 95,137 95,137 0,000 Tree removal : 2907,856 2907,856 2907,856 0,000

[kukougu]

Thank you very much. It can work now.