Started by timer
Running as SYSTEM
Building in workspace /var/lib/jenkins/jobs/tensorflow/workspace
[SSH] script:
module load anaconda3_gpu
module load cudnn
cd python
rm -f tensorflow_test.txt
srun --nodes=1 --ntasks-per-node=1 --mem=16g \
--time=00:05:00 \
--partition=a100rome \
--account=bbmb-hydro \
--gpus-per-node=1 \
--gpu-bind=closest \
python3 cifar10gpu.py | tee tensorflow_test.txt
YVALUE=`grep "wall time seconds" tensorflow_test.txt | cut -d'=' -f2`
echo YVALUE=$YVALUE > tensorflow_time.txt
[SSH] executing...
srun: job 96867 queued and waiting for resources
srun: job 96867 has been allocated resources
2024-11-08 07:43:02.933082: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-08 07:43:03.734592: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-11-08 07:43:06.831646: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-11-08 07:43:06.832999: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /sw/spack/hydrogpu-2022-06/apps/cudnn/8.4.0.27-11.6-gcc-11.3.0-qbhsfav/lib:/sw/spack/hydrogpu-2022-06/apps/cuda/11.7.0-gcc-11.3.0-3ysno6b/lib64:/sw/spack/hydrogpu-2022-06/apps/openmpi/4.1.4-gcc-11.3.0-i6koipj/lib:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib64:/sw/spack/hydrogpu-2022-06/apps/gcc/11.3.0-gcc-8.4.1-kroofxa/lib
2024-11-08 07:43:06.833008: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2024-11-08 07:43:14.174737: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-11-08 07:43:15.088426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1616] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 79148 MB memory: -> device: 0, name: NVIDIA A100 80GB PCIe, pci bus id: 0000:21:00.0, compute capability: 8.0
(50000, 32, 32, 3) (10000, 32, 32, 3)
Epoch 1/10
2024-11-08 07:43:17.380890: I tensorflow/stream_executor/cuda/cuda_blas.cc:1614] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
1/1563 [..............................] - ETA: 29:22 - loss: 2.4344 - accuracy: 0.0938
38/1563 [..............................] - ETA: 2s - loss: 2.2603 - accuracy: 0.1505
76/1563 [>.............................] - ETA: 2s - loss: 2.1924 - accuracy: 0.1850
114/1563 [=>............................] - ETA: 1s - loss: 2.1393 - accuracy: 0.2081
152/1563 [=>............................] - ETA: 1s - loss: 2.1014 - accuracy: 0.2290
191/1563 [==>...........................] - ETA: 1s - loss: 2.0716 - accuracy: 0.2464
230/1563 [===>..........................] - ETA: 1s - loss: 2.0414 - accuracy: 0.2582
268/1563 [====>.........................] - ETA: 1s - loss: 2.0246 - accuracy: 0.2656
307/1563 [====>.........................] - ETA: 1s - loss: 2.0103 - accuracy: 0.2726
345/1563 [=====>........................] - ETA: 1s - loss: 1.9908 - accuracy: 0.2810
383/1563 [======>.......................] - ETA: 1s - loss: 1.9777 - accuracy: 0.2852
422/1563 [=======>......................] - ETA: 1s - loss: 1.9648 - accuracy: 0.2915
460/1563 [=======>......................] - ETA: 1s - loss: 1.9533 - accuracy: 0.2967
498/1563 [========>.....................] - ETA: 1s - loss: 1.9424 - accuracy: 0.3021
536/1563 [=========>....................] - ETA: 1s - loss: 1.9340 - accuracy: 0.3049
573/1563 [=========>....................] - ETA: 1s - loss: 1.9263 - accuracy: 0.3073
610/1563 [==========>...................] - ETA: 1s - loss: 1.9161 - accuracy: 0.3126
648/1563 [===========>..................] - ETA: 1s - loss: 1.9099 - accuracy: 0.3143
686/1563 [============>.................] - ETA: 1s - loss: 1.9025 - accuracy: 0.3181
724/1563 [============>.................] - ETA: 1s - loss: 1.8968 - accuracy: 0.3208
761/1563 [=============>................] - ETA: 1s - loss: 1.8918 - accuracy: 0.3228
799/1563 [==============>...............] - ETA: 1s - loss: 1.8854 - accuracy: 0.3259
837/1563 [===============>..............] - ETA: 0s - loss: 1.8821 - accuracy: 0.3272
875/1563 [===============>..............] - ETA: 0s - loss: 1.8759 - accuracy: 0.3295
913/1563 [================>.............] - ETA: 0s - loss: 1.8721 - accuracy: 0.3313
951/1563 [=================>............] - ETA: 0s - loss: 1.8673 - accuracy: 0.3327
988/1563 [=================>............] - ETA: 0s - loss: 1.8630 - accuracy: 0.3341
1026/1563 [==================>...........] - ETA: 0s - loss: 1.8595 - accuracy: 0.3359
1064/1563 [===================>..........] - ETA: 0s - loss: 1.8549 - accuracy: 0.3373
1101/1563 [====================>.........] - ETA: 0s - loss: 1.8498 - accuracy: 0.3397
1139/1563 [====================>.........] - ETA: 0s - loss: 1.8450 - accuracy: 0.3416
1177/1563 [=====================>........] - ETA: 0s - loss: 1.8417 - accuracy: 0.3428
1215/1563 [======================>.......] - ETA: 0s - loss: 1.8393 - accuracy: 0.3444
1253/1563 [=======================>......] - ETA: 0s - loss: 1.8356 - accuracy: 0.3459
1291/1563 [=======================>......] - ETA: 0s - loss: 1.8312 - accuracy: 0.3472
1329/1563 [========================>.....] - ETA: 0s - loss: 1.8266 - accuracy: 0.3492
1367/1563 [=========================>....] - ETA: 0s - loss: 1.8244 - accuracy: 0.3494
1405/1563 [=========================>....] - ETA: 0s - loss: 1.8213 - accuracy: 0.3506
1443/1563 [==========================>...] - ETA: 0s - loss: 1.8192 - accuracy: 0.3515
1481/1563 [===========================>..] - ETA: 0s - loss: 1.8164 - accuracy: 0.3532
1519/1563 [============================>.] - ETA: 0s - loss: 1.8135 - accuracy: 0.3545
1557/1563 [============================>.] - ETA: 0s - loss: 1.8112 - accuracy: 0.3555
1563/1563 [==============================] - 3s 1ms/step - loss: 1.8108 - accuracy: 0.3557
Epoch 2/10
1/1563 [..............................] - ETA: 2s - loss: 1.5759 - accuracy: 0.2812
39/1563 [..............................] - ETA: 2s - loss: 1.6326 - accuracy: 0.4111
77/1563 [>.............................] - ETA: 1s - loss: 1.6418 - accuracy: 0.4127
115/1563 [=>............................] - ETA: 1s - loss: 1.6628 - accuracy: 0.4090
153/1563 [=>............................] - ETA: 1s - loss: 1.6605 - accuracy: 0.4134
191/1563 [==>...........................] - ETA: 1s - loss: 1.6608 - accuracy: 0.4097
229/1563 [===>..........................] - ETA: 1s - loss: 1.6642 - accuracy: 0.4091
267/1563 [====>.........................] - ETA: 1s - loss: 1.6658 - accuracy: 0.4102
305/1563 [====>.........................] - ETA: 1s - loss: 1.6660 - accuracy: 0.4108
343/1563 [=====>........................] - ETA: 1s - loss: 1.6694 - accuracy: 0.4112
381/1563 [======>.......................] - ETA: 1s - loss: 1.6651 - accuracy: 0.4117
419/1563 [=======>......................] - ETA: 1s - loss: 1.6648 - accuracy: 0.4113
457/1563 [=======>......................] - ETA: 1s - loss: 1.6640 - accuracy: 0.4099
495/1563 [========>.....................] - ETA: 1s - loss: 1.6608 - accuracy: 0.4105
533/1563 [=========>....................] - ETA: 1s - loss: 1.6577 - accuracy: 0.4111
571/1563 [=========>....................] - ETA: 1s - loss: 1.6552 - accuracy: 0.4111
609/1563 [==========>...................] - ETA: 1s - loss: 1.6513 - accuracy: 0.4136
646/1563 [===========>..................] - ETA: 1s - loss: 1.6498 - accuracy: 0.4145
684/1563 [============>.................] - ETA: 1s - loss: 1.6493 - accuracy: 0.4142
722/1563 [============>.................] - ETA: 1s - loss: 1.6482 - accuracy: 0.4151
760/1563 [=============>................] - ETA: 1s - loss: 1.6464 - accuracy: 0.4156
798/1563 [==============>...............] - ETA: 1s - loss: 1.6435 - accuracy: 0.4161
836/1563 [===============>..............] - ETA: 0s - loss: 1.6433 - accuracy: 0.4168
874/1563 [===============>..............] - ETA: 0s - loss: 1.6422 - accuracy: 0.4175
912/1563 [================>.............] - ETA: 0s - loss: 1.6393 - accuracy: 0.4184
950/1563 [=================>............] - ETA: 0s - loss: 1.6389 - accuracy: 0.4189
987/1563 [=================>............] - ETA: 0s - loss: 1.6377 - accuracy: 0.4203
1024/1563 [==================>...........] - ETA: 0s - loss: 1.6368 - accuracy: 0.4207
1062/1563 [===================>..........] - ETA: 0s - loss: 1.6364 - accuracy: 0.4213
1099/1563 [====================>.........] - ETA: 0s - loss: 1.6359 - accuracy: 0.4212
1137/1563 [====================>.........] - ETA: 0s - loss: 1.6346 - accuracy: 0.4219
1175/1563 [=====================>........] - ETA: 0s - loss: 1.6333 - accuracy: 0.4232
1213/1563 [======================>.......] - ETA: 0s - loss: 1.6326 - accuracy: 0.4235
1251/1563 [=======================>......] - ETA: 0s - loss: 1.6305 - accuracy: 0.4239
1288/1563 [=======================>......] - ETA: 0s - loss: 1.6313 - accuracy: 0.4241
1325/1563 [========================>.....] - ETA: 0s - loss: 1.6319 - accuracy: 0.4238
1363/1563 [=========================>....] - ETA: 0s - loss: 1.6304 - accuracy: 0.4240
1401/1563 [=========================>....] - ETA: 0s - loss: 1.6289 - accuracy: 0.4245
1438/1563 [==========================>...] - ETA: 0s - loss: 1.6280 - accuracy: 0.4251
1475/1563 [===========================>..] - ETA: 0s - loss: 1.6261 - accuracy: 0.4258
1513/1563 [============================>.] - ETA: 0s - loss: 1.6246 - accuracy: 0.4259
1551/1563 [============================>.] - ETA: 0s - loss: 1.6238 - accuracy: 0.4262
1563/1563 [==============================] - 2s 1ms/step - loss: 1.6237 - accuracy: 0.4262
Epoch 3/10
1/1563 [..............................] - ETA: 2s - loss: 1.7686 - accuracy: 0.4062
39/1563 [..............................] - ETA: 2s - loss: 1.5215 - accuracy: 0.4527
77/1563 [>.............................] - ETA: 2s - loss: 1.5391 - accuracy: 0.4452
115/1563 [=>............................] - ETA: 1s - loss: 1.5486 - accuracy: 0.4484
153/1563 [=>............................] - ETA: 1s - loss: 1.5647 - accuracy: 0.4434
190/1563 [==>...........................] - ETA: 1s - loss: 1.5543 - accuracy: 0.4502
229/1563 [===>..........................] - ETA: 1s - loss: 1.5557 - accuracy: 0.4499
268/1563 [====>.........................] - ETA: 1s - loss: 1.5533 - accuracy: 0.4509
307/1563 [====>.........................] - ETA: 1s - loss: 1.5540 - accuracy: 0.4512
346/1563 [=====>........................] - ETA: 1s - loss: 1.5522 - accuracy: 0.4523
385/1563 [======>.......................] - ETA: 1s - loss: 1.5526 - accuracy: 0.4512
424/1563 [=======>......................] - ETA: 1s - loss: 1.5561 - accuracy: 0.4506
463/1563 [=======>......................] - ETA: 1s - loss: 1.5522 - accuracy: 0.4519
502/1563 [========>.....................] - ETA: 1s - loss: 1.5572 - accuracy: 0.4496
541/1563 [=========>....................] - ETA: 1s - loss: 1.5593 - accuracy: 0.4496
580/1563 [==========>...................] - ETA: 1s - loss: 1.5566 - accuracy: 0.4524
619/1563 [==========>...................] - ETA: 1s - loss: 1.5567 - accuracy: 0.4526
658/1563 [===========>..................] - ETA: 1s - loss: 1.5577 - accuracy: 0.4517
697/1563 [============>.................] - ETA: 1s - loss: 1.5573 - accuracy: 0.4513
735/1563 [=============>................] - ETA: 1s - loss: 1.5557 - accuracy: 0.4509
774/1563 [=============>................] - ETA: 1s - loss: 1.5537 - accuracy: 0.4510
813/1563 [==============>...............] - ETA: 0s - loss: 1.5513 - accuracy: 0.4518
852/1563 [===============>..............] - ETA: 0s - loss: 1.5488 - accuracy: 0.4529
891/1563 [================>.............] - ETA: 0s - loss: 1.5501 - accuracy: 0.4533
929/1563 [================>.............] - ETA: 0s - loss: 1.5500 - accuracy: 0.4536
968/1563 [=================>............] - ETA: 0s - loss: 1.5506 - accuracy: 0.4539
1007/1563 [==================>...........] - ETA: 0s - loss: 1.5500 - accuracy: 0.4541
1046/1563 [===================>..........] - ETA: 0s - loss: 1.5500 - accuracy: 0.4541
1085/1563 [===================>..........] - ETA: 0s - loss: 1.5500 - accuracy: 0.4544
1124/1563 [====================>.........] - ETA: 0s - loss: 1.5507 - accuracy: 0.4548
1163/1563 [=====================>........] - ETA: 0s - loss: 1.5494 - accuracy: 0.4552
1202/1563 [======================>.......] - ETA: 0s - loss: 1.5490 - accuracy: 0.4554
1241/1563 [======================>.......] - ETA: 0s - loss: 1.5476 - accuracy: 0.4560
1279/1563 [=======================>......] - ETA: 0s - loss: 1.5460 - accuracy: 0.4567
1318/1563 [========================>.....] - ETA: 0s - loss: 1.5463 - accuracy: 0.4572
1356/1563 [=========================>....] - ETA: 0s - loss: 1.5456 - accuracy: 0.4575
1395/1563 [=========================>....] - ETA: 0s - loss: 1.5439 - accuracy: 0.4576
1434/1563 [==========================>...] - ETA: 0s - loss: 1.5425 - accuracy: 0.4583
1473/1563 [===========================>..] - ETA: 0s - loss: 1.5425 - accuracy: 0.4575
1512/1563 [============================>.] - ETA: 0s - loss: 1.5423 - accuracy: 0.4572
1550/1563 [============================>.] - ETA: 0s - loss: 1.5414 - accuracy: 0.4572
1563/1563 [==============================] - 2s 1ms/step - loss: 1.5412 - accuracy: 0.4575
Epoch 4/10
1/1563 [..............................] - ETA: 1s - loss: 1.8522 - accuracy: 0.4062
39/1563 [..............................] - ETA: 2s - loss: 1.5263 - accuracy: 0.4696
78/1563 [>.............................] - ETA: 1s - loss: 1.4818 - accuracy: 0.4800
116/1563 [=>............................] - ETA: 1s - loss: 1.4923 - accuracy: 0.4749
155/1563 [=>............................] - ETA: 1s - loss: 1.4998 - accuracy: 0.4702
193/1563 [==>...........................] - ETA: 1s - loss: 1.5044 - accuracy: 0.4697
231/1563 [===>..........................] - ETA: 1s - loss: 1.5021 - accuracy: 0.4692
269/1563 [====>.........................] - ETA: 1s - loss: 1.5053 - accuracy: 0.4672
307/1563 [====>.........................] - ETA: 1s - loss: 1.5068 - accuracy: 0.4666
345/1563 [=====>........................] - ETA: 1s - loss: 1.5064 - accuracy: 0.4662
383/1563 [======>.......................] - ETA: 1s - loss: 1.5056 - accuracy: 0.4668
422/1563 [=======>......................] - ETA: 1s - loss: 1.5060 - accuracy: 0.4662
461/1563 [=======>......................] - ETA: 1s - loss: 1.5077 - accuracy: 0.4652
500/1563 [========>.....................] - ETA: 1s - loss: 1.5039 - accuracy: 0.4669
539/1563 [=========>....................] - ETA: 1s - loss: 1.5015 - accuracy: 0.4682
577/1563 [==========>...................] - ETA: 1s - loss: 1.4965 - accuracy: 0.4705
616/1563 [==========>...................] - ETA: 1s - loss: 1.4937 - accuracy: 0.4718
654/1563 [===========>..................] - ETA: 1s - loss: 1.4937 - accuracy: 0.4724
692/1563 [============>.................] - ETA: 1s - loss: 1.4928 - accuracy: 0.4737
731/1563 [=============>................] - ETA: 1s - loss: 1.4926 - accuracy: 0.4739
770/1563 [=============>................] - ETA: 1s - loss: 1.4913 - accuracy: 0.4743
809/1563 [==============>...............] - ETA: 0s - loss: 1.4894 - accuracy: 0.4750
847/1563 [===============>..............] - ETA: 0s - loss: 1.4887 - accuracy: 0.4751
885/1563 [===============>..............] - ETA: 0s - loss: 1.4884 - accuracy: 0.4748
923/1563 [================>.............] - ETA: 0s - loss: 1.4882 - accuracy: 0.4757
962/1563 [=================>............] - ETA: 0s - loss: 1.4878 - accuracy: 0.4758
1001/1563 [==================>...........] - ETA: 0s - loss: 1.4869 - accuracy: 0.4764
1040/1563 [==================>...........] - ETA: 0s - loss: 1.4866 - accuracy: 0.4761
1079/1563 [===================>..........] - ETA: 0s - loss: 1.4871 - accuracy: 0.4759
1118/1563 [====================>.........] - ETA: 0s - loss: 1.4880 - accuracy: 0.4757
1157/1563 [=====================>........] - ETA: 0s - loss: 1.4888 - accuracy: 0.4753
1196/1563 [=====================>........] - ETA: 0s - loss: 1.4894 - accuracy: 0.4753
1234/1563 [======================>.......] - ETA: 0s - loss: 1.4877 - accuracy: 0.4756
1273/1563 [=======================>......] - ETA: 0s - loss: 1.4862 - accuracy: 0.4765
1312/1563 [========================>.....] - ETA: 0s - loss: 1.4863 - accuracy: 0.4766
1351/1563 [========================>.....] - ETA: 0s - loss: 1.4867 - accuracy: 0.4769
1390/1563 [=========================>....] - ETA: 0s - loss: 1.4853 - accuracy: 0.4772
1429/1563 [==========================>...] - ETA: 0s - loss: 1.4836 - accuracy: 0.4778
1467/1563 [===========================>..] - ETA: 0s - loss: 1.4835 - accuracy: 0.4777
1505/1563 [===========================>..] - ETA: 0s - loss: 1.4829 - accuracy: 0.4779
1544/1563 [============================>.] - ETA: 0s - loss: 1.4821 - accuracy: 0.4785
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4810 - accuracy: 0.4790
Epoch 5/10
1/1563 [..............................] - ETA: 2s - loss: 1.5072 - accuracy: 0.5312
40/1563 [..............................] - ETA: 1s - loss: 1.4435 - accuracy: 0.4859
79/1563 [>.............................] - ETA: 1s - loss: 1.4515 - accuracy: 0.4877
118/1563 [=>............................] - ETA: 1s - loss: 1.4567 - accuracy: 0.4833
157/1563 [==>...........................] - ETA: 1s - loss: 1.4382 - accuracy: 0.4896
196/1563 [==>...........................] - ETA: 1s - loss: 1.4444 - accuracy: 0.4888
235/1563 [===>..........................] - ETA: 1s - loss: 1.4448 - accuracy: 0.4878
274/1563 [====>.........................] - ETA: 1s - loss: 1.4427 - accuracy: 0.4880
313/1563 [=====>........................] - ETA: 1s - loss: 1.4439 - accuracy: 0.4890
352/1563 [=====>........................] - ETA: 1s - loss: 1.4390 - accuracy: 0.4919
391/1563 [======>.......................] - ETA: 1s - loss: 1.4330 - accuracy: 0.4940
430/1563 [=======>......................] - ETA: 1s - loss: 1.4303 - accuracy: 0.4954
469/1563 [========>.....................] - ETA: 1s - loss: 1.4279 - accuracy: 0.4976
508/1563 [========>.....................] - ETA: 1s - loss: 1.4275 - accuracy: 0.4964
547/1563 [=========>....................] - ETA: 1s - loss: 1.4296 - accuracy: 0.4949
586/1563 [==========>...................] - ETA: 1s - loss: 1.4329 - accuracy: 0.4934
625/1563 [==========>...................] - ETA: 1s - loss: 1.4329 - accuracy: 0.4936
664/1563 [===========>..................] - ETA: 1s - loss: 1.4337 - accuracy: 0.4931
703/1563 [============>.................] - ETA: 1s - loss: 1.4335 - accuracy: 0.4929
742/1563 [=============>................] - ETA: 1s - loss: 1.4345 - accuracy: 0.4926
781/1563 [=============>................] - ETA: 1s - loss: 1.4362 - accuracy: 0.4914
820/1563 [==============>...............] - ETA: 0s - loss: 1.4370 - accuracy: 0.4905
859/1563 [===============>..............] - ETA: 0s - loss: 1.4359 - accuracy: 0.4913
898/1563 [================>.............] - ETA: 0s - loss: 1.4373 - accuracy: 0.4909
937/1563 [================>.............] - ETA: 0s - loss: 1.4386 - accuracy: 0.4906
976/1563 [=================>............] - ETA: 0s - loss: 1.4378 - accuracy: 0.4911
1015/1563 [==================>...........] - ETA: 0s - loss: 1.4363 - accuracy: 0.4920
1054/1563 [===================>..........] - ETA: 0s - loss: 1.4352 - accuracy: 0.4931
1093/1563 [===================>..........] - ETA: 0s - loss: 1.4353 - accuracy: 0.4938
1131/1563 [====================>.........] - ETA: 0s - loss: 1.4354 - accuracy: 0.4939
1169/1563 [=====================>........] - ETA: 0s - loss: 1.4372 - accuracy: 0.4933
1207/1563 [======================>.......] - ETA: 0s - loss: 1.4363 - accuracy: 0.4938
1244/1563 [======================>.......] - ETA: 0s - loss: 1.4353 - accuracy: 0.4945
1281/1563 [=======================>......] - ETA: 0s - loss: 1.4340 - accuracy: 0.4953
1319/1563 [========================>.....] - ETA: 0s - loss: 1.4327 - accuracy: 0.4953
1356/1563 [=========================>....] - ETA: 0s - loss: 1.4339 - accuracy: 0.4953
1394/1563 [=========================>....] - ETA: 0s - loss: 1.4337 - accuracy: 0.4955
1432/1563 [==========================>...] - ETA: 0s - loss: 1.4333 - accuracy: 0.4961
1470/1563 [===========================>..] - ETA: 0s - loss: 1.4321 - accuracy: 0.4962
1508/1563 [===========================>..] - ETA: 0s - loss: 1.4316 - accuracy: 0.4969
1546/1563 [============================>.] - ETA: 0s - loss: 1.4311 - accuracy: 0.4971
1563/1563 [==============================] - 2s 1ms/step - loss: 1.4309 - accuracy: 0.4970
Epoch 6/10
1/1563 [..............................] - ETA: 2s - loss: 1.5350 - accuracy: 0.3750
36/1563 [..............................] - ETA: 2s - loss: 1.4164 - accuracy: 0.5078
73/1563 [>.............................] - ETA: 2s - loss: 1.4337 - accuracy: 0.4953
110/1563 [=>............................] - ETA: 2s - loss: 1.4106 - accuracy: 0.5082
147/1563 [=>............................] - ETA: 1s - loss: 1.4088 - accuracy: 0.5091
185/1563 [==>...........................] - ETA: 1s - loss: 1.3969 - accuracy: 0.5130
222/1563 [===>..........................] - ETA: 1s - loss: 1.3969 - accuracy: 0.5094
260/1563 [===>..........................] - ETA: 1s - loss: 1.3968 - accuracy: 0.5097
298/1563 [====>.........................] - ETA: 1s - loss: 1.4002 - accuracy: 0.5065
335/1563 [=====>........................] - ETA: 1s - loss: 1.4007 - accuracy: 0.5071
373/1563 [======>.......................] - ETA: 1s - loss: 1.3996 - accuracy: 0.5077
411/1563 [======>.......................] - ETA: 1s - loss: 1.3969 - accuracy: 0.5103
449/1563 [=======>......................] - ETA: 1s - loss: 1.3941 - accuracy: 0.5107
487/1563 [========>.....................] - ETA: 1s - loss: 1.3911 - accuracy: 0.5116
525/1563 [=========>....................] - ETA: 1s - loss: 1.3883 - accuracy: 0.5116
563/1563 [=========>....................] - ETA: 1s - loss: 1.3885 - accuracy: 0.5125
600/1563 [==========>...................] - ETA: 1s - loss: 1.3895 - accuracy: 0.5121
637/1563 [===========>..................] - ETA: 1s - loss: 1.3883 - accuracy: 0.5124
674/1563 [===========>..................] - ETA: 1s - loss: 1.3881 - accuracy: 0.5117
711/1563 [============>.................] - ETA: 1s - loss: 1.3889 - accuracy: 0.5118
749/1563 [=============>................] - ETA: 1s - loss: 1.3910 - accuracy: 0.5115
787/1563 [==============>...............] - ETA: 1s - loss: 1.3918 - accuracy: 0.5113
825/1563 [==============>...............] - ETA: 0s - loss: 1.3909 - accuracy: 0.5120
863/1563 [===============>..............] - ETA: 0s - loss: 1.3902 - accuracy: 0.5118
901/1563 [================>.............] - ETA: 0s - loss: 1.3862 - accuracy: 0.5125
939/1563 [=================>............] - ETA: 0s - loss: 1.3866 - accuracy: 0.5115
976/1563 [=================>............] - ETA: 0s - loss: 1.3863 - accuracy: 0.5120
1013/1563 [==================>...........] - ETA: 0s - loss: 1.3872 - accuracy: 0.5119
1050/1563 [===================>..........] - ETA: 0s - loss: 1.3866 - accuracy: 0.5123
1088/1563 [===================>..........] - ETA: 0s - loss: 1.3884 - accuracy: 0.5129
1126/1563 [====================>.........] - ETA: 0s - loss: 1.3867 - accuracy: 0.5134
1164/1563 [=====================>........] - ETA: 0s - loss: 1.3865 - accuracy: 0.5136
1202/1563 [======================>.......] - ETA: 0s - loss: 1.3862 - accuracy: 0.5137
1239/1563 [======================>.......] - ETA: 0s - loss: 1.3863 - accuracy: 0.5138
1277/1563 [=======================>......] - ETA: 0s - loss: 1.3863 - accuracy: 0.5138
1314/1563 [========================>.....] - ETA: 0s - loss: 1.3873 - accuracy: 0.5134
1351/1563 [========================>.....] - ETA: 0s - loss: 1.3870 - accuracy: 0.5132
1388/1563 [=========================>....] - ETA: 0s - loss: 1.3868 - accuracy: 0.5134
1426/1563 [==========================>...] - ETA: 0s - loss: 1.3872 - accuracy: 0.5131
1464/1563 [===========================>..] - ETA: 0s - loss: 1.3862 - accuracy: 0.5133
1502/1563 [===========================>..] - ETA: 0s - loss: 1.3865 - accuracy: 0.5129
1539/1563 [============================>.] - ETA: 0s - loss: 1.3871 - accuracy: 0.5127
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3861 - accuracy: 0.5128
Epoch 7/10
1/1563 [..............................] - ETA: 2s - loss: 1.1675 - accuracy: 0.6250
39/1563 [..............................] - ETA: 2s - loss: 1.3758 - accuracy: 0.5096
78/1563 [>.............................] - ETA: 1s - loss: 1.3607 - accuracy: 0.5244
117/1563 [=>............................] - ETA: 1s - loss: 1.3721 - accuracy: 0.5243
156/1563 [=>............................] - ETA: 1s - loss: 1.3661 - accuracy: 0.5308
194/1563 [==>...........................] - ETA: 1s - loss: 1.3559 - accuracy: 0.5341
233/1563 [===>..........................] - ETA: 1s - loss: 1.3576 - accuracy: 0.5299
268/1563 [====>.........................] - ETA: 1s - loss: 1.3554 - accuracy: 0.5296
306/1563 [====>.........................] - ETA: 1s - loss: 1.3493 - accuracy: 0.5297
343/1563 [=====>........................] - ETA: 1s - loss: 1.3483 - accuracy: 0.5298
380/1563 [======>.......................] - ETA: 1s - loss: 1.3489 - accuracy: 0.5295
418/1563 [=======>......................] - ETA: 1s - loss: 1.3457 - accuracy: 0.5304
456/1563 [=======>......................] - ETA: 1s - loss: 1.3482 - accuracy: 0.5291
494/1563 [========>.....................] - ETA: 1s - loss: 1.3481 - accuracy: 0.5276
532/1563 [=========>....................] - ETA: 1s - loss: 1.3500 - accuracy: 0.5270
570/1563 [=========>....................] - ETA: 1s - loss: 1.3487 - accuracy: 0.5274
608/1563 [==========>...................] - ETA: 1s - loss: 1.3511 - accuracy: 0.5280
645/1563 [===========>..................] - ETA: 1s - loss: 1.3505 - accuracy: 0.5281
682/1563 [============>.................] - ETA: 1s - loss: 1.3524 - accuracy: 0.5278
720/1563 [============>.................] - ETA: 1s - loss: 1.3526 - accuracy: 0.5275
758/1563 [=============>................] - ETA: 1s - loss: 1.3503 - accuracy: 0.5283
796/1563 [==============>...............] - ETA: 1s - loss: 1.3505 - accuracy: 0.5286
834/1563 [===============>..............] - ETA: 0s - loss: 1.3524 - accuracy: 0.5290
871/1563 [===============>..............] - ETA: 0s - loss: 1.3528 - accuracy: 0.5293
908/1563 [================>.............] - ETA: 0s - loss: 1.3549 - accuracy: 0.5283
946/1563 [=================>............] - ETA: 0s - loss: 1.3542 - accuracy: 0.5274
983/1563 [=================>............] - ETA: 0s - loss: 1.3526 - accuracy: 0.5276
1020/1563 [==================>...........] - ETA: 0s - loss: 1.3521 - accuracy: 0.5273
1058/1563 [===================>..........] - ETA: 0s - loss: 1.3532 - accuracy: 0.5258
1097/1563 [====================>.........] - ETA: 0s - loss: 1.3532 - accuracy: 0.5254
1136/1563 [====================>.........] - ETA: 0s - loss: 1.3541 - accuracy: 0.5258
1174/1563 [=====================>........] - ETA: 0s - loss: 1.3529 - accuracy: 0.5259
1210/1563 [======================>.......] - ETA: 0s - loss: 1.3520 - accuracy: 0.5257
1248/1563 [======================>.......] - ETA: 0s - loss: 1.3495 - accuracy: 0.5270
1285/1563 [=======================>......] - ETA: 0s - loss: 1.3503 - accuracy: 0.5271
1322/1563 [========================>.....] - ETA: 0s - loss: 1.3500 - accuracy: 0.5272
1360/1563 [=========================>....] - ETA: 0s - loss: 1.3490 - accuracy: 0.5278
1398/1563 [=========================>....] - ETA: 0s - loss: 1.3489 - accuracy: 0.5282
1436/1563 [==========================>...] - ETA: 0s - loss: 1.3486 - accuracy: 0.5282
1474/1563 [===========================>..] - ETA: 0s - loss: 1.3484 - accuracy: 0.5283
1513/1563 [============================>.] - ETA: 0s - loss: 1.3472 - accuracy: 0.5287
1551/1563 [============================>.] - ETA: 0s - loss: 1.3468 - accuracy: 0.5287
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3473 - accuracy: 0.5285
Epoch 8/10
1/1563 [..............................] - ETA: 2s - loss: 1.6089 - accuracy: 0.3750
39/1563 [..............................] - ETA: 2s - loss: 1.3639 - accuracy: 0.5112
78/1563 [>.............................] - ETA: 1s - loss: 1.3444 - accuracy: 0.5256
117/1563 [=>............................] - ETA: 1s - loss: 1.3263 - accuracy: 0.5366
156/1563 [=>............................] - ETA: 1s - loss: 1.3240 - accuracy: 0.5387
194/1563 [==>...........................] - ETA: 1s - loss: 1.3121 - accuracy: 0.5440
232/1563 [===>..........................] - ETA: 1s - loss: 1.3029 - accuracy: 0.5462
270/1563 [====>.........................] - ETA: 1s - loss: 1.3048 - accuracy: 0.5434
308/1563 [====>.........................] - ETA: 1s - loss: 1.3033 - accuracy: 0.5420
346/1563 [=====>........................] - ETA: 1s - loss: 1.3055 - accuracy: 0.5397
384/1563 [======>.......................] - ETA: 1s - loss: 1.3081 - accuracy: 0.5394
423/1563 [=======>......................] - ETA: 1s - loss: 1.3091 - accuracy: 0.5386
461/1563 [=======>......................] - ETA: 1s - loss: 1.3072 - accuracy: 0.5386
499/1563 [========>.....................] - ETA: 1s - loss: 1.3111 - accuracy: 0.5367
537/1563 [=========>....................] - ETA: 1s - loss: 1.3098 - accuracy: 0.5371
574/1563 [==========>...................] - ETA: 1s - loss: 1.3112 - accuracy: 0.5360
612/1563 [==========>...................] - ETA: 1s - loss: 1.3122 - accuracy: 0.5363
650/1563 [===========>..................] - ETA: 1s - loss: 1.3124 - accuracy: 0.5366
688/1563 [============>.................] - ETA: 1s - loss: 1.3146 - accuracy: 0.5349
727/1563 [============>.................] - ETA: 1s - loss: 1.3149 - accuracy: 0.5347
765/1563 [=============>................] - ETA: 1s - loss: 1.3169 - accuracy: 0.5340
803/1563 [==============>...............] - ETA: 1s - loss: 1.3175 - accuracy: 0.5343
841/1563 [===============>..............] - ETA: 0s - loss: 1.3179 - accuracy: 0.5337
879/1563 [===============>..............] - ETA: 0s - loss: 1.3188 - accuracy: 0.5332
917/1563 [================>.............] - ETA: 0s - loss: 1.3175 - accuracy: 0.5341
955/1563 [=================>............] - ETA: 0s - loss: 1.3201 - accuracy: 0.5339
994/1563 [==================>...........] - ETA: 0s - loss: 1.3199 - accuracy: 0.5343
1033/1563 [==================>...........] - ETA: 0s - loss: 1.3184 - accuracy: 0.5350
1071/1563 [===================>..........] - ETA: 0s - loss: 1.3183 - accuracy: 0.5352
1109/1563 [====================>.........] - ETA: 0s - loss: 1.3197 - accuracy: 0.5348
1147/1563 [=====================>........] - ETA: 0s - loss: 1.3189 - accuracy: 0.5351
1185/1563 [=====================>........] - ETA: 0s - loss: 1.3176 - accuracy: 0.5355
1223/1563 [======================>.......] - ETA: 0s - loss: 1.3170 - accuracy: 0.5359
1261/1563 [=======================>......] - ETA: 0s - loss: 1.3155 - accuracy: 0.5363
1299/1563 [=======================>......] - ETA: 0s - loss: 1.3145 - accuracy: 0.5361
1337/1563 [========================>.....] - ETA: 0s - loss: 1.3153 - accuracy: 0.5360
1375/1563 [=========================>....] - ETA: 0s - loss: 1.3145 - accuracy: 0.5368
1413/1563 [==========================>...] - ETA: 0s - loss: 1.3143 - accuracy: 0.5371
1451/1563 [==========================>...] - ETA: 0s - loss: 1.3145 - accuracy: 0.5374
1489/1563 [===========================>..] - ETA: 0s - loss: 1.3158 - accuracy: 0.5365
1527/1563 [============================>.] - ETA: 0s - loss: 1.3148 - accuracy: 0.5368
1563/1563 [==============================] - 2s 1ms/step - loss: 1.3151 - accuracy: 0.5368
Epoch 9/10
1/1563 [..............................] - ETA: 2s - loss: 1.0482 - accuracy: 0.6250
40/1563 [..............................] - ETA: 2s - loss: 1.2663 - accuracy: 0.5578
79/1563 [>.............................] - ETA: 1s - loss: 1.2440 - accuracy: 0.5589
117/1563 [=>............................] - ETA: 1s - loss: 1.2469 - accuracy: 0.5588
156/1563 [=>............................] - ETA: 1s - loss: 1.2616 - accuracy: 0.5561
194/1563 [==>...........................] - ETA: 1s - loss: 1.2669 - accuracy: 0.5567
232/1563 [===>..........................] - ETA: 1s - loss: 1.2608 - accuracy: 0.5567
270/1563 [====>.........................] - ETA: 1s - loss: 1.2576 - accuracy: 0.5588
309/1563 [====>.........................] - ETA: 1s - loss: 1.2613 - accuracy: 0.5568
348/1563 [=====>........................] - ETA: 1s - loss: 1.2684 - accuracy: 0.5559
387/1563 [======>.......................] - ETA: 1s - loss: 1.2684 - accuracy: 0.5559
425/1563 [=======>......................] - ETA: 1s - loss: 1.2657 - accuracy: 0.5575
463/1563 [=======>......................] - ETA: 1s - loss: 1.2663 - accuracy: 0.5566
501/1563 [========>.....................] - ETA: 1s - loss: 1.2686 - accuracy: 0.5548
539/1563 [=========>....................] - ETA: 1s - loss: 1.2700 - accuracy: 0.5532
577/1563 [==========>...................] - ETA: 1s - loss: 1.2712 - accuracy: 0.5529
615/1563 [==========>...................] - ETA: 1s - loss: 1.2747 - accuracy: 0.5526
654/1563 [===========>..................] - ETA: 1s - loss: 1.2776 - accuracy: 0.5522
692/1563 [============>.................] - ETA: 1s - loss: 1.2750 - accuracy: 0.5534
730/1563 [=============>................] - ETA: 1s - loss: 1.2726 - accuracy: 0.5540
768/1563 [=============>................] - ETA: 1s - loss: 1.2734 - accuracy: 0.5542
806/1563 [==============>...............] - ETA: 1s - loss: 1.2717 - accuracy: 0.5539
844/1563 [===============>..............] - ETA: 0s - loss: 1.2727 - accuracy: 0.5547
882/1563 [===============>..............] - ETA: 0s - loss: 1.2760 - accuracy: 0.5537
920/1563 [================>.............] - ETA: 0s - loss: 1.2760 - accuracy: 0.5536
958/1563 [=================>............] - ETA: 0s - loss: 1.2783 - accuracy: 0.5523
997/1563 [==================>...........] - ETA: 0s - loss: 1.2799 - accuracy: 0.5522
1036/1563 [==================>...........] - ETA: 0s - loss: 1.2799 - accuracy: 0.5522
1074/1563 [===================>..........] - ETA: 0s - loss: 1.2797 - accuracy: 0.5522
1112/1563 [====================>.........] - ETA: 0s - loss: 1.2824 - accuracy: 0.5510
1150/1563 [=====================>........] - ETA: 0s - loss: 1.2831 - accuracy: 0.5507
1188/1563 [=====================>........] - ETA: 0s - loss: 1.2840 - accuracy: 0.5503
1226/1563 [======================>.......] - ETA: 0s - loss: 1.2847 - accuracy: 0.5500
1264/1563 [=======================>......] - ETA: 0s - loss: 1.2840 - accuracy: 0.5501
1302/1563 [=======================>......] - ETA: 0s - loss: 1.2836 - accuracy: 0.5500
1340/1563 [========================>.....] - ETA: 0s - loss: 1.2828 - accuracy: 0.5502
1378/1563 [=========================>....] - ETA: 0s - loss: 1.2821 - accuracy: 0.5504
1416/1563 [==========================>...] - ETA: 0s - loss: 1.2825 - accuracy: 0.5504
1454/1563 [==========================>...] - ETA: 0s - loss: 1.2832 - accuracy: 0.5500
1492/1563 [===========================>..] - ETA: 0s - loss: 1.2824 - accuracy: 0.5502
1530/1563 [============================>.] - ETA: 0s - loss: 1.2814 - accuracy: 0.5507
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2804 - accuracy: 0.5513
Epoch 10/10
1/1563 [..............................] - ETA: 2s - loss: 1.3367 - accuracy: 0.5000
39/1563 [..............................] - ETA: 2s - loss: 1.2237 - accuracy: 0.5697
77/1563 [>.............................] - ETA: 1s - loss: 1.2572 - accuracy: 0.5564
115/1563 [=>............................] - ETA: 1s - loss: 1.2518 - accuracy: 0.5652
153/1563 [=>............................] - ETA: 1s - loss: 1.2472 - accuracy: 0.5666
191/1563 [==>...........................] - ETA: 1s - loss: 1.2560 - accuracy: 0.5661
229/1563 [===>..........................] - ETA: 1s - loss: 1.2511 - accuracy: 0.5655
268/1563 [====>.........................] - ETA: 1s - loss: 1.2530 - accuracy: 0.5652
305/1563 [====>.........................] - ETA: 1s - loss: 1.2562 - accuracy: 0.5648
343/1563 [=====>........................] - ETA: 1s - loss: 1.2565 - accuracy: 0.5640
381/1563 [======>.......................] - ETA: 1s - loss: 1.2573 - accuracy: 0.5640
419/1563 [=======>......................] - ETA: 1s - loss: 1.2569 - accuracy: 0.5643
457/1563 [=======>......................] - ETA: 1s - loss: 1.2608 - accuracy: 0.5624
495/1563 [========>.....................] - ETA: 1s - loss: 1.2624 - accuracy: 0.5612
533/1563 [=========>....................] - ETA: 1s - loss: 1.2628 - accuracy: 0.5613
571/1563 [=========>....................] - ETA: 1s - loss: 1.2657 - accuracy: 0.5611
609/1563 [==========>...................] - ETA: 1s - loss: 1.2618 - accuracy: 0.5618
647/1563 [===========>..................] - ETA: 1s - loss: 1.2605 - accuracy: 0.5625
685/1563 [============>.................] - ETA: 1s - loss: 1.2554 - accuracy: 0.5643
723/1563 [============>.................] - ETA: 1s - loss: 1.2553 - accuracy: 0.5637
761/1563 [=============>................] - ETA: 1s - loss: 1.2545 - accuracy: 0.5635
799/1563 [==============>...............] - ETA: 1s - loss: 1.2516 - accuracy: 0.5646
837/1563 [===============>..............] - ETA: 0s - loss: 1.2506 - accuracy: 0.5653
876/1563 [===============>..............] - ETA: 0s - loss: 1.2510 - accuracy: 0.5637
914/1563 [================>.............] - ETA: 0s - loss: 1.2487 - accuracy: 0.5648
952/1563 [=================>............] - ETA: 0s - loss: 1.2470 - accuracy: 0.5650
990/1563 [==================>...........] - ETA: 0s - loss: 1.2478 - accuracy: 0.5638
1029/1563 [==================>...........] - ETA: 0s - loss: 1.2482 - accuracy: 0.5628
1067/1563 [===================>..........] - ETA: 0s - loss: 1.2504 - accuracy: 0.5619
1106/1563 [====================>.........] - ETA: 0s - loss: 1.2511 - accuracy: 0.5616
1144/1563 [====================>.........] - ETA: 0s - loss: 1.2520 - accuracy: 0.5611
1182/1563 [=====================>........] - ETA: 0s - loss: 1.2536 - accuracy: 0.5610
1220/1563 [======================>.......] - ETA: 0s - loss: 1.2532 - accuracy: 0.5610
1258/1563 [=======================>......] - ETA: 0s - loss: 1.2530 - accuracy: 0.5611
1296/1563 [=======================>......] - ETA: 0s - loss: 1.2533 - accuracy: 0.5609
1334/1563 [========================>.....] - ETA: 0s - loss: 1.2527 - accuracy: 0.5614
1372/1563 [=========================>....] - ETA: 0s - loss: 1.2525 - accuracy: 0.5616
1411/1563 [==========================>...] - ETA: 0s - loss: 1.2517 - accuracy: 0.5620
1449/1563 [==========================>...] - ETA: 0s - loss: 1.2510 - accuracy: 0.5618
1488/1563 [===========================>..] - ETA: 0s - loss: 1.2510 - accuracy: 0.5621
1527/1563 [============================>.] - ETA: 0s - loss: 1.2501 - accuracy: 0.5627
1563/1563 [==============================] - 2s 1ms/step - loss: 1.2499 - accuracy: 0.5631
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
flatten (Flatten) (None, 3072) 0
dense (Dense) (None, 3000) 9219000
dense_1 (Dense) (None, 1000) 3001000
dense_2 (Dense) (None, 10) 10010
=================================================================
Total params: 12,230,010
Trainable params: 12,230,010
Non-trainable params: 0
_________________________________________________________________
None
wall time seconds (without loading test data set)= 24.522833585739136
hydro03
[SSH] completed
[SSH] exit-status: 0
[workspace] $ /bin/sh -xe /tmp/jenkins14165781174237902901.sh
+ scp 'HYDRO_REMOTE:~svchydrojenkins/python/tensorflow_time.txt' /var/lib/jenkins/jobs/tensorflow/workspace
Recording plot data
Saving plot series data from: /var/lib/jenkins/jobs/tensorflow/workspace/tensorflow_time.txt
Finished: SUCCESS
![[Jenkins]](/static/755cd4af/images/svgs/logo.svg){kind=logo}
