Started by timer
Running as SYSTEM
Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace
[SSH] script:
module load anaconda3_gpu
module load cudnn
cd python
rm -f tensorflow_test.txt
srun --nodes=1 --ntasks-per-node=1 --mem=16g \
--time=00:05:00 \
--partition=a100rome \
--account=bbmb-hydro \
--gpus-per-node=1 \
--gpu-bind=closest \
python3 cifar10gpu.py | tee tensorflow_test.txt
YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2`
echo YVALUE=$YVALUE > tensorflow_time.txt
[SSH] executing...
srun: job 96317 queued and waiting for resources
srun: job 96317 has been allocated resources
2024-11-01 07:43:07.780981: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-01 07:43:31.216834: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-01 07:44:04.819936: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-11-01 07:44:04.820513: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-11-01 07:44:04.820522: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-11-01 07:44:57.750820: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-01 07:45:00.439793: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0
(50000, 32, 32, 3) (10000, 32, 32, 3)
Epoch 1/10
2024-11-01 07:45:07.290975: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
1/1563 [..............................] - ETA: 1:43:15 - loss: 2.4395 - accuracy: 0.1250
38/1563 [..............................] - ETA: 2s - loss: 2.2476 - accuracy: 0.1686
76/1563 [>.............................] - ETA: 2s - loss: 2.1806 - accuracy: 0.1974
114/1563 [=>............................] - ETA: 1s - loss: 2.1261 - accuracy: 0.2201
153/1563 [=>............................] - ETA: 1s - loss: 2.0824 - accuracy: 0.2392
191/1563 [==>...........................] - ETA: 1s - loss: 2.0581 - accuracy: 0.2492
230/1563 [===>..........................] - ETA: 1s - loss: 2.0345 - accuracy: 0.2626
268/1563 [====>.........................] - ETA: 1s - loss: 2.0174 - accuracy: 0.2715
307/1563 [====>.........................] - ETA: 1s - loss: 2.0031 - accuracy: 0.2778
346/1563 [=====>........................] - ETA: 1s - loss: 1.9889 - accuracy: 0.2819
385/1563 [======>.......................] - ETA: 1s - loss: 1.9742 - accuracy: 0.2886
424/1563 [=======>......................] - ETA: 1s - loss: 1.9619 - accuracy: 0.2935
463/1563 [=======>......................] - ETA: 1s - loss: 1.9485 - accuracy: 0.2970
502/1563 [========>.....................] - ETA: 1s - loss: 1.9402 - accuracy: 0.3004
541/1563 [=========>....................] - ETA: 1s - loss: 1.9305 - accuracy: 0.3050
580/1563 [==========>...................] - ETA: 1s - loss: 1.9195 - accuracy: 0.3089
619/1563 [==========>...................] - ETA: 1s - loss: 1.9109 - accuracy: 0.3127
658/1563 [===========>..................] - ETA: 1s - loss: 1.9025 - accuracy: 0.3163
697/1563 [============>.................] - ETA: 1s - loss: 1.8957 - accuracy: 0.3198
736/1563 [=============>................] - ETA: 1s - loss: 1.8884 - accuracy: 0.3217
775/1563 [=============>................] - ETA: 1s - loss: 1.8835 - accuracy: 0.3241
814/1563 [==============>...............] - ETA: 0s - loss: 1.8757 - accuracy: 0.3281
853/1563 [===============>..............] - ETA: 0s - loss: 1.8726 - accuracy: 0.3296
892/1563 [================>.............] - ETA: 0s - loss: 1.8680 - accuracy: 0.3311
931/1563 [================>.............] - ETA: 0s - loss: 1.8643 - accuracy: 0.3324
970/1563 [=================>............] - ETA: 0s - loss: 1.8607 - accuracy: 0.3343
1009/1563 [==================>...........] - ETA: 0s - loss: 1.8563 - accuracy: 0.3357
1048/1563 [===================>..........] - ETA: 0s - loss: 1.8514 - accuracy: 0.3378
1087/1563 [===================>..........] - ETA: 0s - loss: 1.8475 - accuracy: 0.3392
1126/1563 [====================>.........] - ETA: 0s - loss: 1.8432 - accuracy: 0.3420
1165/1563 [=====================>........] - ETA: 0s - loss: 1.8405 - accuracy: 0.3434
1204/1563 [======================>.......] - ETA: 0s - loss: 1.8367 - accuracy: 0.3460
1243/1563 [======================>.......] - ETA: 0s - loss: 1.8330 - accuracy: 0.3470
1282/1563 [=======================>......] - ETA: 0s - loss: 1.8288 - accuracy: 0.3483
1321/1563 [========================>.....] - ETA: 0s - loss: 1.8246 - accuracy: 0.3499
1360/1563 [=========================>....] - ETA: 0s - loss: 1.8214 - accuracy: 0.3507
1399/1563 [=========================>....] - ETA: 0s - loss: 1.8179 - accuracy: 0.3518
1438/1563 [==========================>...] - ETA: 0s - loss: 1.8155 - accuracy: 0.3523
1477/1563 [===========================>..] - ETA: 0s - loss: 1.8126 - accuracy: 0.3533
1516/1563 [============================>.] - ETA: 0s - loss: 1.8091 - accuracy: 0.3550
1555/1563 [============================>.] - ETA: 0s - loss: 1.8064 - accuracy: 0.3563
1563/1563 [==============================] - 6s 1ms/step - loss: 1.8061 - accuracy: 0.3565
Epoch 2/10
1/1563 [..............................] - ETA: 2s - loss: 2.2452 - accuracy: 0.2812
40/1563 [..............................] - ETA: 1s - loss: 1.7232 - accuracy: 0.4062
78/1563 [>.............................] - ETA: 1s - loss: 1.6935 - accuracy: 0.4119
117/1563 [=>............................] - ETA: 1s - loss: 1.6750 - accuracy: 0.4129
156/1563 [=>............................] - ETA: 1s - loss: 1.6877 - accuracy: 0.4056
195/1563 [==>...........................] - ETA: 1s - loss: 1.6723 - accuracy: 0.4101
234/1563 [===>..........................] - ETA: 1s - loss: 1.6745 - accuracy: 0.4132
273/1563 [====>.........................] - ETA: 1s - loss: 1.6739 - accuracy: 0.4107
312/1563 [====>.........................] - ETA: 1s - loss: 1.6662 - accuracy: 0.4140
351/1563 [=====>........................] - ETA: 1s - loss: 1.6660 - accuracy: 0.4150
390/1563 [======>.......................] - ETA: 1s - loss: 1.6686 - accuracy: 0.4141
429/1563 [=======>......................] - ETA: 1s - loss: 1.6661 - accuracy: 0.4148
468/1563 [=======>......................] - ETA: 1s - loss: 1.6662 - accuracy: 0.4141
507/1563 [========>.....................] - ETA: 1s - loss: 1.6605 - accuracy: 0.4163
546/1563 [=========>....................] - ETA: 1s - loss: 1.6569 - accuracy: 0.4174
585/1563 [==========>...................] - ETA: 1s - loss: 1.6522 - accuracy: 0.4187
624/1563 [==========>...................] - ETA: 1s - loss: 1.6539 - accuracy: 0.4175
663/1563 [===========>..................] - ETA: 1s - loss: 1.6530 - accuracy: 0.4176
702/1563 [============>.................] - ETA: 1s - loss: 1.6495 - accuracy: 0.4192
741/1563 [=============>................] - ETA: 1s - loss: 1.6487 - accuracy: 0.4200
780/1563 [=============>................] - ETA: 1s - loss: 1.6468 - accuracy: 0.4211
819/1563 [==============>...............] - ETA: 0s - loss: 1.6436 - accuracy: 0.4227
857/1563 [===============>..............] - ETA: 0s - loss: 1.6441 - accuracy: 0.4223
895/1563 [================>.............] - ETA: 0s - loss: 1.6423 - accuracy: 0.4233
934/1563 [================>.............] - ETA: 0s - loss: 1.6421 - accuracy: 0.4239
972/1563 [=================>............] - ETA: 0s - loss: 1.6414 - accuracy: 0.4240
1010/1563 [==================>...........] - ETA: 0s - loss: 1.6397 - accuracy: 0.4245
1048/1563 [===================>..........] - ETA: 0s - loss: 1.6398 - accuracy: 0.4245
1087/1563 [===================>..........] - ETA: 0s - loss: 1.6382 - accuracy: 0.4249
1125/1563 [====================>.........] - ETA: 0s - loss: 1.6377 - accuracy: 0.4250
1164/1563 [=====================>........] - ETA: 0s - loss: 1.6373 - accuracy: 0.4250
1203/1563 [======================>.......] - ETA: 0s - loss: 1.6368 - accuracy: 0.4254
1242/1563 [======================>.......] - ETA: 0s - loss: 1.6353 - accuracy: 0.4258
1280/1563 [=======================>......] - ETA: 0s - loss: 1.6351 - accuracy: 0.4260
1318/1563 [========================>.....] - ETA: 0s - loss: 1.6349 - accuracy: 0.4258
1356/1563 [=========================>....] - ETA: 0s - loss: 1.6336 - accuracy: 0.4259
1394/1563 [=========================>....] - ETA: 0s - loss: 1.6318 - accuracy: 0.4263
1432/1563 [==========================>...] - ETA: 0s - loss: 1.6298 - accuracy: 0.4268
1471/1563 [===========================>..] - ETA: 0s - loss: 1.6285 - accuracy: 0.4274
1509/1563 [===========================>..] - ETA: 0s - loss: 1.6259 - accuracy: 0.4283
1548/1563 [============================>.] - ETA: 0s - loss: 1.6242 - accuracy: 0.4284
1563/1563 [==============================] - 2s 1ms/step - loss: 1.6240 - accuracy: 0.4284
Epoch 3/10
1/1563 [..............................] - ETA: 2s - loss: 1.6127 - accuracy: 0.4062
39/1563 [..............................] - ETA: 2s - loss: 1.5500 - accuracy: 0.4455
78/1563 [>.............................] - ETA: 1s - loss: 1.5528 - accuracy: 0.4403
117/1563 [=>............................] - ETA: 1s - loss: 1.5743 - accuracy: 0.4418
155/1563 [=>............................] - ETA: 1s - loss: 1.5796 - accuracy: 0.4405
194/1563 [==>...........................] - ETA: 1s - loss: 1.5803 - accuracy: 0.4398
232/1563 [===>..........................] - ETA: 1s - loss: 1.5691 - accuracy: 0.4463
270/1563 [====>.........................] - ETA: 1s - loss: 1.5624 - accuracy: 0.4486
308/1563 [====>.........................] - ETA: 1s - loss: 1.5620 - accuracy: 0.4500
346/1563 [=====>........................] - ETA: 1s - loss: 1.5579 - accuracy: 0.4530
385/1563 [======>.......................] - ETA: 1s - loss: 1.5564 - accuracy: 0.4532
424/1563 [=======>......................] - ETA: 1s - loss: 1.5605 - accuracy: 0.4501
463/1563 [=======>......................] - ETA: 1s - loss: 1.5628 - accuracy: 0.4486
502/1563 [========>.....................] - ETA: 1s - loss: 1.5608 - accuracy: 0.4498
541/1563 [=========>....................] - ETA: 1s - loss: 1.5569 - accuracy: 0.4503
580/1563 [==========>...................] - ETA: 1s - loss: 1.5558 - accuracy: 0.4503
619/1563 [==========>...................] - ETA: 1s - loss: 1.5552 - accuracy: 0.4514
657/1563 [===========>..................] - ETA: 1s - loss: 1.5546 - accuracy: 0.4519
695/1563 [============>.................] - ETA: 1s - loss: 1.5530 - accuracy: 0.4527
733/1563 [=============>................] - ETA: 1s - loss: 1.5535 - accuracy: 0.4528
771/1563 [=============>................] - ETA: 1s - loss: 1.5524 - accuracy: 0.4534
810/1563 [==============>...............] - ETA: 0s - loss: 1.5531 - accuracy: 0.4530
849/1563 [===============>..............] - ETA: 0s - loss: 1.5516 - accuracy: 0.4534
888/1563 [================>.............] - ETA: 0s - loss: 1.5538 - accuracy: 0.4528
926/1563 [================>.............] - ETA: 0s - loss: 1.5529 - accuracy: 0.4527
965/1563 [=================>............] - ETA: 0s - loss: 1.5515 - accuracy: 0.4533
1003/1563 [==================>...........] - ETA: 0s - loss: 1.5516 - accuracy: 0.4537
1041/1563 [==================>...........] - ETA: 0s - loss: 1.5508 - accuracy: 0.4539
1079/1563 [===================>..........] - ETA: 0s - loss: 1.5488 - accuracy: 0.4542
1117/1563 [====================>.........] - ETA: 0s - loss: 1.5475 - accuracy: 0.4549
1156/1563 [=====================>........] - ETA: 0s - loss: 1.5462 - accuracy: 0.4553
1195/1563 [=====================>........] - ETA: 0s - loss: 1.5450 - accuracy: 0.4548
1234/1563 [======================>.......] - ETA: 0s - loss: 1.5443 - accuracy: 0.4552
1273/1563 [=======================>......] - ETA: 0s - loss: 1.5446 - accuracy: 0.4551
1312/1563 [========================>.....] - ETA: 0s - loss: 1.5443 - accuracy: 0.4552
1349/1563 [========================>.....] - ETA: 0s - loss: 1.5431 - accuracy: 0.4558
1387/1563 [=========================>....] - ETA: 0s - loss: 1.5426 - accuracy: 0.4559
1426/1563 [==========================>...] - ETA: 0s - loss: 1.5426 - accuracy: 0.4564
1465/1563 [===========================>..] - ETA: 0s - loss: 1.5425 - accuracy: 0.4564
1503/1563 [===========================>..] - ETA: 0s - loss: 1.5425 - accuracy: 0.4568
1542/1563 [============================>.] - ETA: 0s - loss: 1.5416 - accuracy: 0.4573
1563/1563 [==============================] - 2s 1ms/step - loss: 1.5422 - accuracy: 0.4571
Epoch 4/10
1/1563 [..............................] - ETA: 2s - loss: 1.6679 - accuracy: 0.4375
39/1563 [..............................] - ETA: 2s - loss: 1.4869 - accuracy: 0.4744
77/1563 [>.............................] - ETA: 1s - loss: 1.5256 - accuracy: 0.4582
116/1563 [=>............................] - ETA: 1s - loss: 1.5075 - accuracy: 0.4671
155/1563 [=>............................] - ETA: 1s - loss: 1.4893 - accuracy: 0.4770
194/1563 [==>...........................] - ETA: 1s - loss: 1.4980 - accuracy: 0.4702
232/1563 [===>..........................] - ETA: 1s - loss: 1.4943 - accuracy: 0.4709
271/1563 [====>.........................] - ETA: 1s - loss: 1.4899 - accuracy: 0.4730
309/1563 [====>.........................] - ETA: 1s - loss: 1.4897 - accuracy: 0.4730
347/1563 [=====>........................] - ETA: 1s - loss: 1.4941 - accuracy: 0.4713
385/1563 [======>.......................] - ETA: 1s - loss: 1.4968 - accuracy: 0.4722
424/1563 [=======>......................] - ETA: 1s - loss: 1.4974 - accuracy: 0.4731
462/1563 [=======>......................] - ETA: 1s - loss: 1.4966 - accuracy: 0.4738
500/1563 [========>.....................] - ETA: 1s - loss: 1.4968 - accuracy: 0.4748
539/1563 [=========>....................] - ETA: 1s - loss: 1.4939 - accuracy: 0.4761
577/1563 [==========>...................] - ETA: 1s - loss: 1.4936 - accuracy: 0.4767
616/1563 [==========>...................] - ETA: 1s - loss: 1.4900 - accuracy: 0.4778
654/1563 [===========>..................] - ETA: 1s - loss: 1.4901 - accuracy: 0.4774
692/1563 [============>.................] - ETA: 1s - loss: 1.4893 - accuracy: 0.4773
730/1563 [=============>................] - ETA: 1s - loss: 1.4913 - accuracy: 0.4765
769/1563 [=============>................] - ETA: 1s - loss: 1.4911 - accuracy: 0.4764
808/1563 [==============>...............] - ETA: 0s - loss: 1.4887 - accuracy: 0.4776
847/1563 [===============>..............] - ETA: 0s - loss: 1.4861 - accuracy: 0.4781
886/1563 [================>.............] - ETA: 0s - loss: 1.4861 - accuracy: 0.4775
925/1563 [================>.............] - ETA: 0s - loss: 1.4855 - accuracy: 0.4773
963/1563 [=================>............] - ETA: 0s - loss: 1.4850 - accuracy: 0.4776
1001/1563 [==================>...........] - ETA: 0s - loss: 1.4834 - accuracy: 0.4786
1039/1563 [==================>...........] - ETA: 0s - loss: 1.4828 - accuracy: 0.4792
1078/1563 [===================>..........] - ETA: 0s - loss: 1.4823 - accuracy: 0.4791
1117/1563 [====================>.........] - ETA: 0s - loss: 1.4821 - accuracy: 0.4787
1156/1563 [=====================>........] - ETA: 0s - loss: 1.4827 - accuracy: 0.4782
1195/1563 [=====================>........] - ETA: 0s - loss: 1.4838 - accuracy: 0.4776
1233/1563 [======================>.......] - ETA: 0s - loss: 1.4827 - accuracy: 0.4781
1271/1563 [=======================>......] - ETA: 0s - loss: 1.4817 - accuracy: 0.4785
1309/1563 [========================>.....] - ETA: 0s - loss: 1.4823 - accuracy: 0.4788
1348/1563 [========================>.....] - ETA: 0s - loss: 1.4828 - accuracy: 0.4787
1387/1563 [=========================>....] - ETA: 0s - loss: 1.4839 - accuracy: 0.4785
1426/1563 [==========================>...] - ETA: 0s - loss: 1.4843 - accuracy: 0.4783
1465/1563 [===========================>..] - ETA: 0s - loss: 1.4841 - accuracy: 0.4780
1504/1563 [===========================>..] - ETA: 0s - loss: 1.4834 - accuracy: 0.4783
1542/1563 [============================>.] - ETA: 0s - loss: 1.4837 - accuracy: 0.4781
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4830 - accuracy: 0.4779
Epoch 5/10
1/1563 [..............................] - ETA: 1s - loss: 1.3014 - accuracy: 0.5625
39/1563 [..............................] - ETA: 2s - loss: 1.4907 - accuracy: 0.4776
78/1563 [>.............................] - ETA: 1s - loss: 1.4549 - accuracy: 0.4896
117/1563 [=>............................] - ETA: 1s - loss: 1.4510 - accuracy: 0.4861
156/1563 [=>............................] - ETA: 1s - loss: 1.4349 - accuracy: 0.4938
195/1563 [==>...........................] - ETA: 1s - loss: 1.4418 - accuracy: 0.4912
233/1563 [===>..........................] - ETA: 1s - loss: 1.4490 - accuracy: 0.4882
271/1563 [====>.........................] - ETA: 1s - loss: 1.4485 - accuracy: 0.4890
309/1563 [====>.........................] - ETA: 1s - loss: 1.4456 - accuracy: 0.4908
347/1563 [=====>........................] - ETA: 1s - loss: 1.4432 - accuracy: 0.4934
385/1563 [======>.......................] - ETA: 1s - loss: 1.4410 - accuracy: 0.4933
422/1563 [=======>......................] - ETA: 1s - loss: 1.4417 - accuracy: 0.4926
461/1563 [=======>......................] - ETA: 1s - loss: 1.4418 - accuracy: 0.4929
500/1563 [========>.....................] - ETA: 1s - loss: 1.4423 - accuracy: 0.4936
539/1563 [=========>....................] - ETA: 1s - loss: 1.4460 - accuracy: 0.4917
577/1563 [==========>...................] - ETA: 1s - loss: 1.4462 - accuracy: 0.4928
616/1563 [==========>...................] - ETA: 1s - loss: 1.4423 - accuracy: 0.4934
655/1563 [===========>..................] - ETA: 1s - loss: 1.4411 - accuracy: 0.4938
694/1563 [============>.................] - ETA: 1s - loss: 1.4437 - accuracy: 0.4933
733/1563 [=============>................] - ETA: 1s - loss: 1.4433 - accuracy: 0.4929
772/1563 [=============>................] - ETA: 1s - loss: 1.4439 - accuracy: 0.4921
811/1563 [==============>...............] - ETA: 0s - loss: 1.4407 - accuracy: 0.4933
850/1563 [===============>..............] - ETA: 0s - loss: 1.4379 - accuracy: 0.4937
889/1563 [================>.............] - ETA: 0s - loss: 1.4376 - accuracy: 0.4938
928/1563 [================>.............] - ETA: 0s - loss: 1.4392 - accuracy: 0.4927
967/1563 [=================>............] - ETA: 0s - loss: 1.4400 - accuracy: 0.4923
1006/1563 [==================>...........] - ETA: 0s - loss: 1.4393 - accuracy: 0.4932
1044/1563 [===================>..........] - ETA: 0s - loss: 1.4375 - accuracy: 0.4941
1083/1563 [===================>..........] - ETA: 0s - loss: 1.4384 - accuracy: 0.4942
1122/1563 [====================>.........] - ETA: 0s - loss: 1.4381 - accuracy: 0.4948
1161/1563 [=====================>........] - ETA: 0s - loss: 1.4386 - accuracy: 0.4946
1200/1563 [======================>.......] - ETA: 0s - loss: 1.4383 - accuracy: 0.4947
1239/1563 [======================>.......] - ETA: 0s - loss: 1.4385 - accuracy: 0.4947
1278/1563 [=======================>......] - ETA: 0s - loss: 1.4390 - accuracy: 0.4945
1317/1563 [========================>.....] - ETA: 0s - loss: 1.4380 - accuracy: 0.4946
1355/1563 [=========================>....] - ETA: 0s - loss: 1.4380 - accuracy: 0.4945
1393/1563 [=========================>....] - ETA: 0s - loss: 1.4379 - accuracy: 0.4948
1432/1563 [==========================>...] - ETA: 0s - loss: 1.4382 - accuracy: 0.4946
1470/1563 [===========================>..] - ETA: 0s - loss: 1.4374 - accuracy: 0.4946
1509/1563 [===========================>..] - ETA: 0s - loss: 1.4355 - accuracy: 0.4951
1547/1563 [============================>.] - ETA: 0s - loss: 1.4341 - accuracy: 0.4952
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4334 - accuracy: 0.4954
Epoch 6/10
1/1563 [..............................] - ETA: 2s - loss: 1.3326 - accuracy: 0.5625
39/1563 [..............................] - ETA: 2s - loss: 1.4283 - accuracy: 0.4880
78/1563 [>.............................] - ETA: 1s - loss: 1.3962 - accuracy: 0.5024
117/1563 [=>............................] - ETA: 1s - loss: 1.4035 - accuracy: 0.5021
156/1563 [=>............................] - ETA: 1s - loss: 1.3971 - accuracy: 0.5066
195/1563 [==>...........................] - ETA: 1s - loss: 1.3905 - accuracy: 0.5111
234/1563 [===>..........................] - ETA: 1s - loss: 1.3943 - accuracy: 0.5097
273/1563 [====>.........................] - ETA: 1s - loss: 1.3935 - accuracy: 0.5127
312/1563 [====>.........................] - ETA: 1s - loss: 1.3901 - accuracy: 0.5133
351/1563 [=====>........................] - ETA: 1s - loss: 1.3884 - accuracy: 0.5148
390/1563 [======>.......................] - ETA: 1s - loss: 1.3891 - accuracy: 0.5154
429/1563 [=======>......................] - ETA: 1s - loss: 1.3880 - accuracy: 0.5152
468/1563 [=======>......................] - ETA: 1s - loss: 1.3913 - accuracy: 0.5142
506/1563 [========>.....................] - ETA: 1s - loss: 1.3940 - accuracy: 0.5134
544/1563 [=========>....................] - ETA: 1s - loss: 1.3933 - accuracy: 0.5124
583/1563 [==========>...................] - ETA: 1s - loss: 1.3914 - accuracy: 0.5133
621/1563 [==========>...................] - ETA: 1s - loss: 1.3930 - accuracy: 0.5125
659/1563 [===========>..................] - ETA: 1s - loss: 1.3936 - accuracy: 0.5126
697/1563 [============>.................] - ETA: 1s - loss: 1.3942 - accuracy: 0.5121
735/1563 [=============>................] - ETA: 1s - loss: 1.3929 - accuracy: 0.5123
773/1563 [=============>................] - ETA: 1s - loss: 1.3932 - accuracy: 0.5122
812/1563 [==============>...............] - ETA: 0s - loss: 1.3938 - accuracy: 0.5120
851/1563 [===============>..............] - ETA: 0s - loss: 1.3933 - accuracy: 0.5118
890/1563 [================>.............] - ETA: 0s - loss: 1.3944 - accuracy: 0.5110
929/1563 [================>.............] - ETA: 0s - loss: 1.3918 - accuracy: 0.5125
968/1563 [=================>............] - ETA: 0s - loss: 1.3912 - accuracy: 0.5126
1006/1563 [==================>...........] - ETA: 0s - loss: 1.3889 - accuracy: 0.5128
1045/1563 [===================>..........] - ETA: 0s - loss: 1.3895 - accuracy: 0.5124
1084/1563 [===================>..........] - ETA: 0s - loss: 1.3884 - accuracy: 0.5128
1123/1563 [====================>.........] - ETA: 0s - loss: 1.3872 - accuracy: 0.5128
1162/1563 [=====================>........] - ETA: 0s - loss: 1.3871 - accuracy: 0.5126
1201/1563 [======================>.......] - ETA: 0s - loss: 1.3876 - accuracy: 0.5129
1239/1563 [======================>.......] - ETA: 0s - loss: 1.3862 - accuracy: 0.5136
1277/1563 [=======================>......] - ETA: 0s - loss: 1.3868 - accuracy: 0.5134
1316/1563 [========================>.....] - ETA: 0s - loss: 1.3870 - accuracy: 0.5132
1354/1563 [========================>.....] - ETA: 0s - loss: 1.3877 - accuracy: 0.5127
1393/1563 [=========================>....] - ETA: 0s - loss: 1.3871 - accuracy: 0.5126
1432/1563 [==========================>...] - ETA: 0s - loss: 1.3886 - accuracy: 0.5120
1471/1563 [===========================>..] - ETA: 0s - loss: 1.3890 - accuracy: 0.5121
1509/1563 [===========================>..] - ETA: 0s - loss: 1.3895 - accuracy: 0.5120
1548/1563 [============================>.] - ETA: 0s - loss: 1.3885 - accuracy: 0.5127
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3890 - accuracy: 0.5124
Epoch 7/10
1/1563 [..............................] - ETA: 2s - loss: 1.3187 - accuracy: 0.4375
39/1563 [..............................] - ETA: 2s - loss: 1.3758 - accuracy: 0.5192
77/1563 [>.............................] - ETA: 1s - loss: 1.3632 - accuracy: 0.5284
116/1563 [=>............................] - ETA: 1s - loss: 1.3668 - accuracy: 0.5242
155/1563 [=>............................] - ETA: 1s - loss: 1.3652 - accuracy: 0.5244
194/1563 [==>...........................] - ETA: 1s - loss: 1.3650 - accuracy: 0.5240
233/1563 [===>..........................] - ETA: 1s - loss: 1.3592 - accuracy: 0.5264
271/1563 [====>.........................] - ETA: 1s - loss: 1.3573 - accuracy: 0.5261
309/1563 [====>.........................] - ETA: 1s - loss: 1.3562 - accuracy: 0.5254
348/1563 [=====>........................] - ETA: 1s - loss: 1.3537 - accuracy: 0.5265
387/1563 [======>.......................] - ETA: 1s - loss: 1.3507 - accuracy: 0.5265
426/1563 [=======>......................] - ETA: 1s - loss: 1.3527 - accuracy: 0.5259
465/1563 [=======>......................] - ETA: 1s - loss: 1.3536 - accuracy: 0.5261
504/1563 [========>.....................] - ETA: 1s - loss: 1.3519 - accuracy: 0.5276
543/1563 [=========>....................] - ETA: 1s - loss: 1.3512 - accuracy: 0.5275
582/1563 [==========>...................] - ETA: 1s - loss: 1.3518 - accuracy: 0.5270
621/1563 [==========>...................] - ETA: 1s - loss: 1.3537 - accuracy: 0.5256
660/1563 [===========>..................] - ETA: 1s - loss: 1.3558 - accuracy: 0.5248
699/1563 [============>.................] - ETA: 1s - loss: 1.3521 - accuracy: 0.5254
738/1563 [=============>................] - ETA: 1s - loss: 1.3554 - accuracy: 0.5241
777/1563 [=============>................] - ETA: 1s - loss: 1.3550 - accuracy: 0.5247
816/1563 [==============>...............] - ETA: 0s - loss: 1.3571 - accuracy: 0.5239
855/1563 [===============>..............] - ETA: 0s - loss: 1.3574 - accuracy: 0.5246
893/1563 [================>.............] - ETA: 0s - loss: 1.3592 - accuracy: 0.5238
931/1563 [================>.............] - ETA: 0s - loss: 1.3584 - accuracy: 0.5244
970/1563 [=================>............] - ETA: 0s - loss: 1.3553 - accuracy: 0.5246
1009/1563 [==================>...........] - ETA: 0s - loss: 1.3552 - accuracy: 0.5249
1048/1563 [===================>..........] - ETA: 0s - loss: 1.3526 - accuracy: 0.5257
1087/1563 [===================>..........] - ETA: 0s - loss: 1.3521 - accuracy: 0.5259
1126/1563 [====================>.........] - ETA: 0s - loss: 1.3525 - accuracy: 0.5254
1165/1563 [=====================>........] - ETA: 0s - loss: 1.3525 - accuracy: 0.5252
1204/1563 [======================>.......] - ETA: 0s - loss: 1.3522 - accuracy: 0.5249
1243/1563 [======================>.......] - ETA: 0s - loss: 1.3523 - accuracy: 0.5251
1282/1563 [=======================>......] - ETA: 0s - loss: 1.3519 - accuracy: 0.5255
1321/1563 [========================>.....] - ETA: 0s - loss: 1.3518 - accuracy: 0.5256
1360/1563 [=========================>....] - ETA: 0s - loss: 1.3514 - accuracy: 0.5256
1399/1563 [=========================>....] - ETA: 0s - loss: 1.3515 - accuracy: 0.5258
1438/1563 [==========================>...] - ETA: 0s - loss: 1.3513 - accuracy: 0.5264
1477/1563 [===========================>..] - ETA: 0s - loss: 1.3506 - accuracy: 0.5270
1516/1563 [============================>.] - ETA: 0s - loss: 1.3499 - accuracy: 0.5271
1555/1563 [============================>.] - ETA: 0s - loss: 1.3507 - accuracy: 0.5268
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3513 - accuracy: 0.5266
Epoch 8/10
1/1563 [..............................] - ETA: 2s - loss: 1.4238 - accuracy: 0.5000
39/1563 [..............................] - ETA: 2s - loss: 1.2738 - accuracy: 0.5689
78/1563 [>.............................] - ETA: 1s - loss: 1.2916 - accuracy: 0.5505
117/1563 [=>............................] - ETA: 1s - loss: 1.3239 - accuracy: 0.5358
156/1563 [=>............................] - ETA: 1s - loss: 1.3218 - accuracy: 0.5371
195/1563 [==>...........................] - ETA: 1s - loss: 1.3234 - accuracy: 0.5362
234/1563 [===>..........................] - ETA: 1s - loss: 1.3208 - accuracy: 0.5371
273/1563 [====>.........................] - ETA: 1s - loss: 1.3169 - accuracy: 0.5394
312/1563 [====>.........................] - ETA: 1s - loss: 1.3177 - accuracy: 0.5395
351/1563 [=====>........................] - ETA: 1s - loss: 1.3127 - accuracy: 0.5424
390/1563 [======>.......................] - ETA: 1s - loss: 1.3073 - accuracy: 0.5441
429/1563 [=======>......................] - ETA: 1s - loss: 1.3124 - accuracy: 0.5415
468/1563 [=======>......................] - ETA: 1s - loss: 1.3157 - accuracy: 0.5393
507/1563 [========>.....................] - ETA: 1s - loss: 1.3179 - accuracy: 0.5378
546/1563 [=========>....................] - ETA: 1s - loss: 1.3172 - accuracy: 0.5374
584/1563 [==========>...................] - ETA: 1s - loss: 1.3155 - accuracy: 0.5390
623/1563 [==========>...................] - ETA: 1s - loss: 1.3166 - accuracy: 0.5393
662/1563 [===========>..................] - ETA: 1s - loss: 1.3197 - accuracy: 0.5382
701/1563 [============>.................] - ETA: 1s - loss: 1.3191 - accuracy: 0.5388
740/1563 [=============>................] - ETA: 1s - loss: 1.3215 - accuracy: 0.5389
779/1563 [=============>................] - ETA: 1s - loss: 1.3203 - accuracy: 0.5393
818/1563 [==============>...............] - ETA: 0s - loss: 1.3184 - accuracy: 0.5399
857/1563 [===============>..............] - ETA: 0s - loss: 1.3154 - accuracy: 0.5411
896/1563 [================>.............] - ETA: 0s - loss: 1.3177 - accuracy: 0.5399
935/1563 [================>.............] - ETA: 0s - loss: 1.3170 - accuracy: 0.5402
974/1563 [=================>............] - ETA: 0s - loss: 1.3150 - accuracy: 0.5409
1013/1563 [==================>...........] - ETA: 0s - loss: 1.3162 - accuracy: 0.5412
1052/1563 [===================>..........] - ETA: 0s - loss: 1.3148 - accuracy: 0.5417
1091/1563 [===================>..........] - ETA: 0s - loss: 1.3150 - accuracy: 0.5416
1130/1563 [====================>.........] - ETA: 0s - loss: 1.3164 - accuracy: 0.5412
1168/1563 [=====================>........] - ETA: 0s - loss: 1.3162 - accuracy: 0.5407
1206/1563 [======================>.......] - ETA: 0s - loss: 1.3161 - accuracy: 0.5406
1245/1563 [======================>.......] - ETA: 0s - loss: 1.3174 - accuracy: 0.5401
1284/1563 [=======================>......] - ETA: 0s - loss: 1.3181 - accuracy: 0.5398
1323/1563 [========================>.....] - ETA: 0s - loss: 1.3190 - accuracy: 0.5394
1362/1563 [=========================>....] - ETA: 0s - loss: 1.3185 - accuracy: 0.5393
1401/1563 [=========================>....] - ETA: 0s - loss: 1.3182 - accuracy: 0.5392
1440/1563 [==========================>...] - ETA: 0s - loss: 1.3183 - accuracy: 0.5387
1478/1563 [===========================>..] - ETA: 0s - loss: 1.3173 - accuracy: 0.5386
1516/1563 [============================>.] - ETA: 0s - loss: 1.3174 - accuracy: 0.5380
1555/1563 [============================>.] - ETA: 0s - loss: 1.3162 - accuracy: 0.5381
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3163 - accuracy: 0.5382
Epoch 9/10
1/1563 [..............................] - ETA: 2s - loss: 1.2726 - accuracy: 0.5938
40/1563 [..............................] - ETA: 2s - loss: 1.2463 - accuracy: 0.5656
79/1563 [>.............................] - ETA: 1s - loss: 1.2539 - accuracy: 0.5578
118/1563 [=>............................] - ETA: 1s - loss: 1.2510 - accuracy: 0.5644
156/1563 [=>............................] - ETA: 1s - loss: 1.2569 - accuracy: 0.5639
195/1563 [==>...........................] - ETA: 1s - loss: 1.2572 - accuracy: 0.5611
234/1563 [===>..........................] - ETA: 1s - loss: 1.2549 - accuracy: 0.5621
273/1563 [====>.........................] - ETA: 1s - loss: 1.2545 - accuracy: 0.5619
312/1563 [====>.........................] - ETA: 1s - loss: 1.2644 - accuracy: 0.5577
351/1563 [=====>........................] - ETA: 1s - loss: 1.2678 - accuracy: 0.5572
390/1563 [======>.......................] - ETA: 1s - loss: 1.2692 - accuracy: 0.5553
429/1563 [=======>......................] - ETA: 1s - loss: 1.2699 - accuracy: 0.5558
467/1563 [=======>......................] - ETA: 1s - loss: 1.2746 - accuracy: 0.5537
506/1563 [========>.....................] - ETA: 1s - loss: 1.2761 - accuracy: 0.5548
545/1563 [=========>....................] - ETA: 1s - loss: 1.2752 - accuracy: 0.5548
584/1563 [==========>...................] - ETA: 1s - loss: 1.2770 - accuracy: 0.5539
622/1563 [==========>...................] - ETA: 1s - loss: 1.2771 - accuracy: 0.5539
661/1563 [===========>..................] - ETA: 1s - loss: 1.2766 - accuracy: 0.5536
700/1563 [============>.................] - ETA: 1s - loss: 1.2770 - accuracy: 0.5533
739/1563 [=============>................] - ETA: 1s - loss: 1.2775 - accuracy: 0.5523
778/1563 [=============>................] - ETA: 1s - loss: 1.2786 - accuracy: 0.5511
817/1563 [==============>...............] - ETA: 0s - loss: 1.2790 - accuracy: 0.5505
855/1563 [===============>..............] - ETA: 0s - loss: 1.2790 - accuracy: 0.5503
893/1563 [================>.............] - ETA: 0s - loss: 1.2796 - accuracy: 0.5511
932/1563 [================>.............] - ETA: 0s - loss: 1.2799 - accuracy: 0.5510
971/1563 [=================>............] - ETA: 0s - loss: 1.2797 - accuracy: 0.5517
1010/1563 [==================>...........] - ETA: 0s - loss: 1.2823 - accuracy: 0.5511
1049/1563 [===================>..........] - ETA: 0s - loss: 1.2820 - accuracy: 0.5517
1087/1563 [===================>..........] - ETA: 0s - loss: 1.2823 - accuracy: 0.5513
1126/1563 [====================>.........] - ETA: 0s - loss: 1.2827 - accuracy: 0.5509
1165/1563 [=====================>........] - ETA: 0s - loss: 1.2829 - accuracy: 0.5505
1203/1563 [======================>.......] - ETA: 0s - loss: 1.2823 - accuracy: 0.5508
1242/1563 [======================>.......] - ETA: 0s - loss: 1.2840 - accuracy: 0.5503
1281/1563 [=======================>......] - ETA: 0s - loss: 1.2832 - accuracy: 0.5506
1320/1563 [========================>.....] - ETA: 0s - loss: 1.2838 - accuracy: 0.5509
1358/1563 [=========================>....] - ETA: 0s - loss: 1.2827 - accuracy: 0.5516
1397/1563 [=========================>....] - ETA: 0s - loss: 1.2832 - accuracy: 0.5510
1436/1563 [==========================>...] - ETA: 0s - loss: 1.2837 - accuracy: 0.5508
1474/1563 [===========================>..] - ETA: 0s - loss: 1.2854 - accuracy: 0.5497
1512/1563 [============================>.] - ETA: 0s - loss: 1.2860 - accuracy: 0.5493
1551/1563 [============================>.] - ETA: 0s - loss: 1.2849 - accuracy: 0.5499
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2848 - accuracy: 0.5501
Epoch 10/10
1/1563 [..............................] - ETA: 2s - loss: 0.9229 - accuracy: 0.7188
40/1563 [..............................] - ETA: 1s - loss: 1.2499 - accuracy: 0.5656
79/1563 [>.............................] - ETA: 1s - loss: 1.2363 - accuracy: 0.5696
117/1563 [=>............................] - ETA: 1s - loss: 1.2323 - accuracy: 0.5697
155/1563 [=>............................] - ETA: 1s - loss: 1.2298 - accuracy: 0.5710
193/1563 [==>...........................] - ETA: 1s - loss: 1.2425 - accuracy: 0.5693
232/1563 [===>..........................] - ETA: 1s - loss: 1.2501 - accuracy: 0.5657
271/1563 [====>.........................] - ETA: 1s - loss: 1.2470 - accuracy: 0.5638
310/1563 [====>.........................] - ETA: 1s - loss: 1.2494 - accuracy: 0.5633
348/1563 [=====>........................] - ETA: 1s - loss: 1.2498 - accuracy: 0.5629
387/1563 [======>.......................] - ETA: 1s - loss: 1.2468 - accuracy: 0.5639
426/1563 [=======>......................] - ETA: 1s - loss: 1.2468 - accuracy: 0.5659
464/1563 [=======>......................] - ETA: 1s - loss: 1.2497 - accuracy: 0.5644
503/1563 [========>.....................] - ETA: 1s - loss: 1.2482 - accuracy: 0.5655
541/1563 [=========>....................] - ETA: 1s - loss: 1.2513 - accuracy: 0.5641
579/1563 [==========>...................] - ETA: 1s - loss: 1.2506 - accuracy: 0.5647
618/1563 [==========>...................] - ETA: 1s - loss: 1.2514 - accuracy: 0.5645
657/1563 [===========>..................] - ETA: 1s - loss: 1.2541 - accuracy: 0.5628
696/1563 [============>.................] - ETA: 1s - loss: 1.2531 - accuracy: 0.5638
734/1563 [=============>................] - ETA: 1s - loss: 1.2542 - accuracy: 0.5624
773/1563 [=============>................] - ETA: 1s - loss: 1.2534 - accuracy: 0.5622
812/1563 [==============>...............] - ETA: 0s - loss: 1.2531 - accuracy: 0.5628
851/1563 [===============>..............] - ETA: 0s - loss: 1.2550 - accuracy: 0.5622
890/1563 [================>.............] - ETA: 0s - loss: 1.2524 - accuracy: 0.5624
928/1563 [================>.............] - ETA: 0s - loss: 1.2511 - accuracy: 0.5631
967/1563 [=================>............] - ETA: 0s - loss: 1.2523 - accuracy: 0.5623
1006/1563 [==================>...........] - ETA: 0s - loss: 1.2525 - accuracy: 0.5627
1044/1563 [===================>..........] - ETA: 0s - loss: 1.2543 - accuracy: 0.5619
1083/1563 [===================>..........] - ETA: 0s - loss: 1.2540 - accuracy: 0.5624
1121/1563 [====================>.........] - ETA: 0s - loss: 1.2512 - accuracy: 0.5632
1160/1563 [=====================>........] - ETA: 0s - loss: 1.2526 - accuracy: 0.5628
1199/1563 [======================>.......] - ETA: 0s - loss: 1.2535 - accuracy: 0.5624
1237/1563 [======================>.......] - ETA: 0s - loss: 1.2530 - accuracy: 0.5624
1276/1563 [=======================>......] - ETA: 0s - loss: 1.2530 - accuracy: 0.5625
1314/1563 [========================>.....] - ETA: 0s - loss: 1.2536 - accuracy: 0.5621
1353/1563 [========================>.....] - ETA: 0s - loss: 1.2532 - accuracy: 0.5624
1392/1563 [=========================>....] - ETA: 0s - loss: 1.2533 - accuracy: 0.5631
1431/1563 [==========================>...] - ETA: 0s - loss: 1.2538 - accuracy: 0.5630
1470/1563 [===========================>..] - ETA: 0s - loss: 1.2548 - accuracy: 0.5629
1509/1563 [===========================>..] - ETA: 0s - loss: 1.2541 - accuracy: 0.5635
1548/1563 [============================>.] - ETA: 0s - loss: 1.2543 - accuracy: 0.5629
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2543 - accuracy: 0.5628
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten (Flatten) (None, 3072) 0
dense (Dense) (None, 3000) 9219000
dense_1 (Dense) (None, 1000) 3001000
dense_2 (Dense) (None, 10) 10010
=================================================================
Total params: 12,230,010
Trainable params: 12,230,010
Non-trainable params: 0
_________________________________________________________________
None
wall time seconds (without loading test data set)= 36.35249137878418
hydro04
[SSH] completed
[SSH] exit-status: 0
[workspace] $ /bin/sh -xe /tmp/jenkins8731355674684206994.sh
+ scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace
Recording plot data
Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt
Finished: SUCCESS
![[Jenkins]](/static/755cd4af/images/svgs/logo.svg){kind=logo}
